ee046202_tutorial_04_classic_statistics_hypothesis_testing_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/dusk/64/000000/artificial-intelligence.png\" style=\"height:50px;display:inline\"> EE 046202 - Technion - Unsupervised Learning & Data Analysis\n",
    "---\n",
    "\n",
    "#### <a href=\"https://taldatech.github.io\">Tal Daniel</a>\n",
    "\n",
    "## Tutorial 04 - Classical Methods in Statistical Inference - Hypothesis Testing 2\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/100/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [Recap](#-Recap)\n",
    "* [T-Test](#-T-Test-(Hypothesis-Testing-of-Gaussian-with-Unknown-Mean-and-Variance))\n",
    "    * [ Exercise - Statistics for Machine Learning](#-Exercise---Statistics-for-Machine-Learning)\n",
    "* [Pearsons's Chi-Squared ($\\chi^2$) Test](#-Pearson's-Chi-Squared-($\\chi^2$)-Test)\n",
    "* [Uniformly Most Powerful (UMP) Test](#-Uniformly-Most-Powerful-Test-(UMP))\n",
    "* [Which Statistical Test To Use???](#-Which-Statistical-Test-To-Use???)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy.stats import t as t_dist\n",
    "from scipy.stats import chi2\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/rewind.png\" style=\"height:50px;display:inline\"> Recap\n",
    "---\n",
    "In the previous tutorial we described the framework of Hypothesis Testing. Following is a recap of some of the definitions and a brief summmary:\n",
    "\n",
    "* **1. Null & Alternative Hypotheses** -  Formulate the null hypothesis $H_0$: $\\theta \\in \\Theta_0$ (that the observations are the result of pure chance) and the alternative hypothesis $H_1$: $\\theta \\in \\Theta_1$ (that the observations show a real effect combined with a component of chance variation).\n",
    "* **2. Test Statistic** - Identify a test statistic that can be used to assess the truth of the null hypothesis. It is a value computed from sample data. The test statistic is used to assess the strength of evidence in support of a null hypothesis.\n",
    "    * A **statistic** is a real-valued function of the data. For example, the sample mean: $W(X_1, X_2,..., X_n)= \\frac{X_1 + X_2 + ... + X_n}{n}$ is a statistic.\n",
    "    * A **test satistic** is a statistic on we which we build our test.\n",
    "    * **Acceptence Region** $A$ - A *set* $A \\subset \\mathbb{R}$ is defined to be the set of all possible values of the test statistic for which we accept $H_0$.\n",
    "    * **Rejection Region** $R$ - A *set*  $R = \\mathbb{R} - A$ is defined to be the set of all possible values of the test statistic for which we reject $H_0$ and accept $H_1$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **3. P-value & Interpretation** - Compute the P-value, which is the probability that a test statistic, at least as significant as the one observed, would be obtained assuming that the null hypothesis were true. The smaller the P-value, the stronger the evidence **against** the null hypothesis.\n",
    "\n",
    "* **4. Significance Level** - Compare the p-value to an acceptable significance value $\\alpha$ (sometimes called an $\\alpha$ value, a probability threshold below which the null hypothesis will be rejected. Common values are 5% and 1%.). If $p \\leq \\alpha$ (the observed effect is statistically significant), the null hypothesis is ruled out, and the alternative hypothesis is valid.\n",
    "    * Commonly used p-values:\n",
    "\n",
    "|<center> P-Value &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </center> |<center> Wording </center>|\n",
    "| -----------------------------------------| --------------- |\n",
    "|<center> $p > 0.05$ </center>| <center> Not Significant </center>|\n",
    "|<center> $0.01 \\leq p \\leq 0.05$ </center>|<center> Significant</center>|\n",
    "|<center> $0.001 \\leq p < 0.01$</center>| <center> Very Significant </center>|\n",
    "|<center> $p < 0.001$</center>| <center> Extremely Significant </center>|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Error Types\n",
    "---\n",
    "\n",
    "\n",
    "* **Type I Error (False Positive)** -  the incorrect rejection of a true null hypothesis. Usually a type I error leads one to conclude that a supposed effect or relationship exists when in fact it doesn't. \n",
    "    * For example, a test that shows a patient to have a disease when in fact the patient does not have the disease, a fire alarm going on indicating a fire when in fact there is no fire, or an experiment indicating that a medical treatment should cure a disease when in fact it does not.\n",
    "    * The chance of **rejecting the null hypthesis $H_0$, when it is TRUE**, denoted by $\\alpha$\n",
    "    * $\\rightarrow$ the chance of **accepting the null hypthesis $H_0$, when it is TRUE** is $1 - \\alpha$\n",
    "    * Formally:\n",
    "        * Denote a *test statistic* as $W$\n",
    "        * $P(\\text{Type 1 Error} | \\theta) = P(\\text{Reject } H_0|\\theta) = P(W \\in R | \\theta), \\theta \\in \\Theta_0$\n",
    "            * If $P(\\text{Type 1 Error} | \\theta) \\leq \\alpha, \\forall \\theta \\in \\Theta_0$, we say that the test has **significance level** $\\alpha$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "   * **Type II Error (False Negative)** - the failure to reject a false null hypothesis.\n",
    "      * For example, a blood test failing to detect the disease it was designed to detect, in a patient who really has the disease; a fire breaking out and the fire alarm does not ring; or a clinical trial of a medical treatment failing to show that the treatment works when really it does.\n",
    "      * The chance of **not rejecting the null hypothesis $H_0$, when it is FALSE**, denoted by $\\beta$\n",
    "      * $\\rightarrow$ the chance of **rejecting the null hypthesis $H_0$, when it is FALSE** is $1 - \\beta$ (also called **power**)\n",
    "      * Since the alternative hypothesis, $H_1$, usually contains more than one value of $\\theta$, the probability of type II error is usually a **function of $\\theta$**, and denoted by $\\beta$.\n",
    "      * Formally: $\\beta(\\theta) = P(\\text{Accept } H_0 | \\theta), \\text{ for } \\theta \\in \\Theta_1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "#### When To Use What:\n",
    "\n",
    "* 2-sided hypothesis testing for the mean: $H_0: \\mu=\\mu_0, H_1: \\mu \\neq \\mu_0$\n",
    "\n",
    "| <center>Case </center>  | Test Statistic | Acceptance Region |\n",
    "| ------ | --- | --- |\n",
    "| $X_i \\sim \\mathcal{N}(\\mu, \\sigma)$, $\\sigma$ **known** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| $W = \\frac{\\overline{X}  -\\mu_0}{\\frac{\\sigma}{\\sqrt{n}}}$| $\\mid W \\mid \\leq z_{\\frac{\\alpha}{2}} $  |\n",
    "|$n$ large, $X_i$ non-normal &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| $W = \\frac{\\overline{X}  -\\mu_0}{\\frac{S}{\\sqrt{n}}}$| $\\mid W\\mid \\leq z_{\\frac{\\alpha}{2}} $  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* 1-sided hypothesis testing for the mean: $H_0: \\mu \\leq \\mu_0, H_1: \\mu > \\mu_0$\n",
    "\n",
    "| <center>Case </center>  | Test Statistic | Acceptance Region |\n",
    "| ------ | --- | --- |\n",
    "| $X_i \\sim \\mathcal{N}(\\mu, \\sigma)$, $\\sigma$ **known** &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| $W = \\frac{\\overline{X}  -\\mu_0}{\\frac{\\sigma}{\\sqrt{n}}}$| $W \\leq z_{\\alpha} $  |\n",
    "|$n$ large, $X_i$ non-normal &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;| $W = \\frac{\\overline{X}  -\\mu_0}{\\frac{S}{\\sqrt{n}}}$| $W \\leq z_{\\alpha} $  |\n",
    "\n",
    "* The only difference is the *absolute* sign on $W$.\n",
    "\n",
    "All expansions can be found <a href=\"https://www.probabilitycourse.com/chapter8/8_4_3_hypothesis_testing_for_mean.php\">HERE</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/dusk/64/000000/task.png\" style=\"height:30px;display:inline\"> Exercise - How Much Protein Does a Tuna Fish Contain?\n",
    "---\n",
    "* Note: this is for all of you who work out :)\n",
    "\n",
    "Assume that the amount of protein in a Tuna-can is a Gaussian random variable with std $\\sigma=2.5$ grams (that is, the variance is *known*) and *unknown* expectation $\\mu$. We are interested in the following hypothesis: $$ H_0: \\mu=30 $$ $$ H_1 : \\mu=32 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "1. The following test is proposed: take 16 random cans (samples are i.i.d.), if the mean is greater than 30.8 grams, reject $H_0$. What is the probability of type 1 error? What is the probability of type 2 error?\n",
    "2. We decided to use a different test: reject $H_0$ in case $\\overline{X}_n > 31.3$. Can you say if type 1 or 2 error probabilities will increase or decrease without further computations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/dusk/64/000000/idea.png\" style=\"height:30px;display:inline\"> Solution\n",
    "---\n",
    "#### Section 1\n",
    "---\n",
    "#### Q: The following test is proposed: take 16 random cans (samples are i.i.d.), if the mean is greater than 30.8 grams, reject $H_0$. What is the probability of type 1 error? What is the probability of type 2 error?\n",
    "---\n",
    "Since each sample is Gaussian, and the samples are i.i.d., the empirical average $\\overline{X}_n$ of 16 samples is also Gaussian, with the **same** mean, and std of $\\frac{\\sigma}{\\sqrt{16}}$: $$ \\overline{X}_n \\sim \\mathcal{N} (\\mu, \\frac{2.5^2}{16}) $$ (Z-test)\n",
    "\n",
    "* **Type-1-error** (False Positive) $ = P_{H_0}(\\overline{X}_n > 30.8) =\\alpha$ $$ = P(\\frac{\\overline{X}_n-\\mu_{H_0}}{2.5 / \\sqrt{16}} > \\frac{30.8 - \\mu_{H_0}}{2.5 / \\sqrt{16}})$$ $$ = P(\\frac{\\overline{X}_n-30}{2.5 / \\sqrt{16}} > \\frac{30.8 - 30}{2.5 / \\sqrt{16}})$$ $$ = P(Z>1.28) = 1 - P(Z <1.28) = 1 - \\Phi(1.28) = 0.1 $$\n",
    "    * $\\Phi(x) = \\frac{1}{\\sqrt{2\\pi}}\\int_{-\\infty}^x \\exp(\\frac{-x^2}{2})dx$\n",
    "    * <a href=\"https://en.wikipedia.org/wiki/Standard_normal_table#Cumulative\">Comulative distribution function of a standard normal distribution</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Type-2-error** (False Negative) $=P_{H_1}(\\overline{X}_n \\leq 30.8)$ $$ =P(\\frac{\\overline{X}_n-\\mu_{H_1}}{2.5 / \\sqrt{16}} \\leq \\frac{30.8 - \\mu_{H_1}}{2.5 / \\sqrt{16}}) $$ $$ = P(\\frac{\\overline{X}_n-32}{2.5 / \\sqrt{16}} \\leq \\frac{30.8 - 32}{2.5 / \\sqrt{16}}) $$ $$ = P(Z \\geq 1.92) = 1 -  P(Z \\leq 1.92) = 1 - \\Phi(1.92) = 0.0274 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Section 2    \n",
    "---\n",
    "#### Q: We decided to use a different test: reject $H_0$ in case $\\overline{X}_n > 31.3$. Can you say if type 1 or 2 error probabilities will increase or decrease without further computations?\n",
    "---\n",
    "We increased the threshold, such that it less likely to reject $H_0$. That is, the event in which $H_0$ is true, and we reject it is more rare. This means that type-1 error must *decrease*, i.e, we will make less False-Positive mistakes. However, the event in which we accept $H_0$ while $H_1$ is true is **more likely** (notice the trade-off between the error types). This means that the type-2 error must increase, and we will make more False-Negative mistakes.\n",
    "This demnostrates the inherent **trade-off** between type-1 and type-2 errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/t.png\" style=\"height:50px;display:inline\"> T-Test (Hypothesis Testing of Gaussian with Unknown Mean and Variance)\n",
    "---\n",
    "* The $t$-test deals with the scenario where both the mean and variance are unknown.\n",
    "    * It is possible to use $z$-test, however, it is sometimes better to do $t$-test for this case.\n",
    "* It is based on **student-t** distribution, which is a heavy-tail distribution.\n",
    "    * Note that although the student-t distribution may look like a Gaussian, it is different (in the limit of degrees of freedom $\\to \\infty$, student-t approaches Normal distribution). \n",
    "* Assume we are given data $\\{X_i\\}_{i=1}^N$, where $X_i \\sim \\mathcal{N}(\\mu, \\sigma^2)$. Both $\\mu$ and $\\sigma^2$ are unknown.\n",
    "* Let the empirical mean and variance be: $$ \\overline{X}_N = \\frac{1}{N}\\sum_{i=1}^N X_i $$ $$ \\overline{\\sigma}_N^2 = \\frac{1}{N-1} \\sum_{i=1}^N (X_i -\\overline{X}_N)^2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Student-t PDF:\n",
    "\n",
    "<img src=\"./assets/student_t.png\" style=\"height:250px\">\n",
    "\n",
    "* Image Credits: <a href=\"https://commons.wikimedia.org/w/index.php?curid=9546828\">By Skbkekas - Own work, CC BY 3.0</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Assume we wish to test the following 2-sided null and alternative hypotheses given the data: $$ H_0: \\mu = \\mu' $$ $$ H_1 : \\mu \\neq \\mu' $$\n",
    "* The $t$-test statistic is defined by: $$ t=\\frac{\\overline{X}_N - \\mu'}{ \\frac{\\overline{\\sigma}_N}{\\sqrt{N}} } $$\n",
    "    * The $t$-test is $|t| < C$, that is, we **reject** $H_0$ if $|t| \\geq C$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In general, there are three possible alternative hypotheses and rejection regions:\n",
    "\n",
    "|**Alternative Hytpothesis** | **Rejection Region**|\n",
    "|----------------------------|---------------------|\n",
    "|<center>$H_a: \\mu_1 \\neq \\mu_2$</center>    | <center>$\\mid T \\mid > t_{n-1, 1-\\frac{\\alpha}{2}}$ </center>|\n",
    "|<center>$H_a: \\mu_1 > \\mu_2$</center>    | <center>$T > t_{n-1, 1-\\alpha}$ </center>|\n",
    "|<center>$H_a: \\mu_1 < \\mu_2$</center>    | <center>$T < t_{n-1, \\alpha}$ </center>|\n",
    "\n",
    "<a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm\">More Details</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/dusk/64/000000/task.png\" style=\"height:30px;display:inline\"> Exercise - Statistics for Machine Learning\n",
    "---\n",
    "In this exercise we will use the $t$-test to estimate the performance of two classifiers on a single domain. The data is taken from \"Evaluating Learning Algorithms - A Classification Perspective\", N.Japkowicz, M.Shah.\n",
    "* **Preview**: Two data samples are matched if they come from repeated observation of the same subject. Given two types of samples, we want to test whether the difference in means between these two samples is *significant*. One way of doing so is by looking at the difference in observed means and standard deviations (the first and second moments of the samples) between these two samples.\n",
    "* In **classification** problems, we often try out different classifiers (which are the output of different learning algorithms) and check their performance. Eventually, we wish to choose the best classifier (the one with the lowest classification error)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Consider two classifiers and their performance on the data. **How can we tell if one is better than the other?**\n",
    "* The null hypothesis, $H_0$, will be that the classifiers are equivalent and we will see if we can reject $H_0$.\n",
    "* Let $\\mathcal{X}, \\mathcal{Y}$ be the feature space and the label space accordingly. Let $f: \\mathcal{X} \\to \\mathcal{Y}$ be a classifier. Denote the validation data (the data on which is not seen during training, but is used to tune the hyper-parameters of the algorithms) as $\\mathcal{D} = \\{x_i, y_i\\}_{i=1}^n$.\n",
    "* Define our **performance measure** (pm): $$ pm_i(f) = \\begin{cases} 1, \\quad f \\text{ classified incorrectly } x_i \\\\ 0, \\quad \\text{otherwise} \\end{cases} $$ \n",
    "    * Note that it is actually the error (\"...**incorrectly**...\").\n",
    "* The *average* performance measure of $f$: $$ \\overline{pm}(f) = \\frac{1}{n}\\sum_{i=1}^n pm_i(f) $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Performance *difference* measures of two classifier $f_1$ and $f_2$: $$ d_i = pm_i(f_1) - pm_i(f_2) $$ $$ \\overline{d} = \\overline{pm}(f_1) - \\overline{pm}(f_2) $$\n",
    "* Define the t-statistics (as we do not know the mean nor the std): $$ t = \\frac{\\overline{d} - 0}{\\tilde{\\sigma}_d / \\sqrt{n}} $$\n",
    "    * $\\tilde{\\sigma}_d^2 = \\frac{1}{n-1} \\sum_{i=1}^n (d_i -  \\overline{d})^2$\n",
    "    * Recall that our null hypothesis, $H_0$ is that the classifiers are equivalent, thus, the expected difference between their performance mesures is 0.\n",
    "    \n",
    "* The data: performance measures of 10 cross-validation sets on some given data. The algorithms were repeated for 10 trials, and the performance was measured in each case. The values are *emprical risks* - the error.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Trial No.</th>\n",
       "      <th>Classifiers</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>f9</th>\n",
       "      <th>f10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>4</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>5</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>5</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>5</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>6</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.6667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>6</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>7</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>7</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>7</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>8</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>8</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>8</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>9</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>9</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>9</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>10</td>\n",
       "      <td>c4.5</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>10</td>\n",
       "      <td>RIP</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>10</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Trial No. Classifiers      f1      f2      f3      f4      f5      f6  \\\n",
       "0           1        c4.5  0.5000  0.0000  0.3333  0.0000  0.3333  0.3333   \n",
       "1           1         RIP  0.3333  0.5000  0.0000  0.0000  0.3333  0.5000   \n",
       "2           1          NB  0.1667  0.0000  0.0000  0.0000  0.0000  0.1667   \n",
       "3           2        c4.5  0.3333  0.1667  0.1667  0.3333  0.3333  0.0000   \n",
       "4           2         RIP  0.3333  0.1667  0.0000  0.0000  0.3333  0.1667   \n",
       "5           2          NB  0.3333  0.0000  0.1667  0.1667  0.0000  0.0000   \n",
       "6           3        c4.5  0.0000  0.0000  0.3333  0.3333  0.1667  0.3333   \n",
       "7           3         RIP  0.0000  0.0000  0.1667  0.3333  0.1667  0.1667   \n",
       "8           3          NB  0.0000  0.0000  0.0000  0.0000  0.0000  0.1667   \n",
       "9           4        c4.5  0.5000  0.1667  0.3333  0.1667  0.0000  0.1667   \n",
       "10          4         RIP  0.3333  0.0000  0.1667  0.1667  0.1667  0.0000   \n",
       "11          4          NB  0.3333  0.0000  0.0000  0.1667  0.0000  0.0000   \n",
       "12          5        c4.5  0.0000  0.1667  0.3333  0.1667  0.0000  0.1667   \n",
       "13          5         RIP  0.3333  0.1667  0.1667  0.1667  0.0000  0.1667   \n",
       "14          5          NB  0.0000  0.1667  0.0000  0.0000  0.1667  0.0000   \n",
       "15          6        c4.5  0.3333  0.1667  0.1667  0.1667  0.6667  0.1667   \n",
       "16          6         RIP  0.0000  0.0000  0.3333  0.0000  0.5000  0.0000   \n",
       "17          6          NB  0.0000  0.0000  0.1667  0.0000  0.1667  0.1667   \n",
       "18          7        c4.5  0.1667  0.3333  0.1667  0.3333  0.1667  0.1667   \n",
       "19          7         RIP  0.1667  0.1667  0.1667  0.3333  0.1667  0.1667   \n",
       "20          7          NB  0.1667  0.1667  0.1667  0.0000  0.1667  0.3333   \n",
       "21          8        c4.5  0.0000  0.1667  0.1667  0.0000  0.0000  0.0000   \n",
       "22          8         RIP  0.1667  0.1667  0.1667  0.0000  0.0000  0.1667   \n",
       "23          8          NB  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "24          9        c4.5  0.1667  0.3333  0.1667  0.3333  0.3333  0.3333   \n",
       "25          9         RIP  0.0000  0.0000  0.1667  0.0000  0.0000  0.1667   \n",
       "26          9          NB  0.0000  0.0000  0.0000  0.1667  0.0000  0.0000   \n",
       "27         10        c4.5  0.5000  0.1667  0.1667  0.5000  0.5000  0.3333   \n",
       "28         10         RIP  0.5000  0.1667  0.1667  0.1667  0.0000  0.3333   \n",
       "29         10          NB  0.5000  0.1667  0.0000  0.0000  0.0000  0.0000   \n",
       "\n",
       "        f7   f8   f9  f10  \n",
       "0   0.3333  0.2  0.2  0.2  \n",
       "1   0.1667  0.0  0.2  0.0  \n",
       "2   0.0000  0.4  0.0  0.0  \n",
       "3   0.0000  0.0  0.2  0.2  \n",
       "4   0.0000  0.2  0.2  0.0  \n",
       "5   0.0000  0.0  0.0  0.0  \n",
       "6   0.1667  0.0  0.2  0.2  \n",
       "7   0.1667  0.2  0.4  0.2  \n",
       "8   0.0000  0.0  0.0  0.0  \n",
       "9   0.5000  0.2  0.2  0.4  \n",
       "10  0.6667  0.2  0.2  0.2  \n",
       "11  0.0000  0.0  0.0  0.2  \n",
       "12  0.0000  0.2  0.4  0.2  \n",
       "13  0.0000  0.4  0.0  0.2  \n",
       "14  0.0000  0.0  0.0  0.4  \n",
       "15  0.3333  0.2  0.2  0.0  \n",
       "16  0.0000  0.0  0.2  0.0  \n",
       "17  0.0000  0.0  0.0  0.0  \n",
       "18  0.3333  0.2  0.0  0.2  \n",
       "19  0.3333  0.4  0.0  0.2  \n",
       "20  0.3333  0.0  0.2  0.0  \n",
       "21  0.1667  0.0  0.6  0.4  \n",
       "22  0.0000  0.0  0.2  0.2  \n",
       "23  0.0000  0.0  0.2  0.2  \n",
       "24  0.0000  0.6  0.2  0.2  \n",
       "25  0.0000  0.4  0.0  0.2  \n",
       "26  0.0000  0.0  0.0  0.2  \n",
       "27  0.0000  0.4  0.4  0.0  \n",
       "28  0.3333  0.2  0.2  0.2  \n",
       "29  0.1667  0.2  0.2  0.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's see the data\n",
    "data = pd.read_csv('./datasets/ml_classifiers.csv')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Trial No.</th>\n",
       "      <th>Classifiers</th>\n",
       "      <th>f1</th>\n",
       "      <th>f2</th>\n",
       "      <th>f3</th>\n",
       "      <th>f4</th>\n",
       "      <th>f5</th>\n",
       "      <th>f6</th>\n",
       "      <th>f7</th>\n",
       "      <th>f8</th>\n",
       "      <th>f9</th>\n",
       "      <th>f10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>8.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>10.0</td>\n",
       "      <td>NB</td>\n",
       "      <td>0.5000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Trial No. Classifiers      f1      f2      f3      f4      f5      f6  \\\n",
       "2         1.0          NB  0.1667  0.0000  0.0000  0.0000  0.0000  0.1667   \n",
       "5         2.0          NB  0.3333  0.0000  0.1667  0.1667  0.0000  0.0000   \n",
       "8         3.0          NB  0.0000  0.0000  0.0000  0.0000  0.0000  0.1667   \n",
       "11        4.0          NB  0.3333  0.0000  0.0000  0.1667  0.0000  0.0000   \n",
       "14        5.0          NB  0.0000  0.1667  0.0000  0.0000  0.1667  0.0000   \n",
       "17        6.0          NB  0.0000  0.0000  0.1667  0.0000  0.1667  0.1667   \n",
       "20        7.0          NB  0.1667  0.1667  0.1667  0.0000  0.1667  0.3333   \n",
       "23        8.0          NB  0.0000  0.0000  0.0000  0.0000  0.0000  0.0000   \n",
       "26        9.0          NB  0.0000  0.0000  0.0000  0.1667  0.0000  0.0000   \n",
       "29       10.0          NB  0.5000  0.1667  0.0000  0.0000  0.0000  0.0000   \n",
       "\n",
       "        f7   f8   f9  f10  \n",
       "2   0.0000  0.4  0.0  0.0  \n",
       "5   0.0000  0.0  0.0  0.0  \n",
       "8   0.0000  0.0  0.0  0.0  \n",
       "11  0.0000  0.0  0.0  0.2  \n",
       "14  0.0000  0.0  0.0  0.4  \n",
       "17  0.0000  0.0  0.0  0.0  \n",
       "20  0.3333  0.0  0.2  0.0  \n",
       "23  0.0000  0.0  0.2  0.2  \n",
       "26  0.0000  0.0  0.0  0.2  \n",
       "29  0.1667  0.2  0.2  0.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for a certain classifier:\n",
    "data.where(data['Classifiers'] == 'NB').dropna()  # without dropna(), it will show NaN on other rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Questions\n",
    "---\n",
    "1. Map the problem to the framework of Hypothesis Testing, with a test in the form of a t-test.\n",
    "2. Define $d_i$ as the *average* difference per trial. Calculate $d_i$ between the classifier c4.5 and NB for every one of the 10 given trials.\n",
    "3. Calculate $\\overline{d}, \\tilde{\\sigma}_d^2$ and $t$.\n",
    "4. For the significane levels of $\\alpha=0.05$, do the two classifiers have similar performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### <img src=\"https://img.icons8.com/dusk/64/000000/idea.png\" style=\"height:30px;display:inline\"> Solution\n",
    "---\n",
    "#### Section 1\n",
    "---\n",
    "#### Q: Map the problem to the framework of Hypothesis Testing, with a test in the form of a t-test.\n",
    "---\n",
    "* We ask a question about the *random variable* $d_i$. \n",
    "* $d_i$ measures the performance difference of the algorithms when they are trained on a specific realization of the data (which is an RV). \n",
    "* Our null hypothesis, $H_0$, is that  $\\mu_{d_i} = 0$: the two classifiers are similar. \n",
    "* However, we do not know the variance of the random variable. \n",
    "* We can assume that: $$ d_i \\sim \\mathcal{N} (\\mu', \\sigma) $$ where the parameters are unknown. \n",
    "* This assumption makes sense in the *limit of large data* and under the (wrong) assumption that each entry of the table is drawn independently from a Gaussian distribution. \n",
    "* Since we do not know the variance, this framework exactly fits the **t-test** framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Section 2\n",
    "---\n",
    "#### Q: Define $d_i$ as the *average* difference per trial. Calculate $d_i$ between the classifier c4.5 and NB for every one of the 10 given trials.\n",
    "---\n",
    "* Recall that $d_i$ is the difference in performance measure between the two classifiers. \n",
    "* We are given 10 partitions of the dataset (for cross-validation) and we also perform 10 trials. \n",
    "* Thus, $d_i$ is the difference of performance in the $i^{th}$ trial, and we calculate it as the average over cross-validations sets. $$ d_i = \\frac{1}{10} \\sum_{j=1}^{10} pm_{i,j} (c4.5) -pm_{i,j} (NB), $$\n",
    "where $j$ is the index of the cross-valdiation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "differnce in performance measure for c4.5 and NB:\n",
      "d_1 = 0.16998000000000005\n",
      "d_2 = 0.10665999999999999\n",
      "d_3 = 0.15666\n",
      "d_4 = 0.19334\n",
      "d_5 = 0.08999999999999998\n",
      "d_6 = 0.19\n",
      "d_7 = 0.053330000000000016\n",
      "d_8 = 0.11000999999999997\n",
      "d_9 = 0.22999\n",
      "d_10 = 0.17333\n"
     ]
    }
   ],
   "source": [
    "# let's calculate all of the d_i\n",
    "nb_data = data.where(data['Classifiers'] == 'NB').dropna().drop(columns=['Trial No.', 'Classifiers']).mean(axis=1).values\n",
    "c45_data = data.where(data['Classifiers'] == 'c4.5').dropna().drop(columns=['Trial No.', 'Classifiers']).mean(axis=1).values\n",
    "d = c45_data - nb_data\n",
    "print(\"differnce in performance measure for c4.5 and NB:\")\n",
    "for i in range(len(d)):\n",
    "    print(\"d_{} = {}\".format(i + 1, d[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Section 3\n",
    "---\n",
    "#### Q: Calculate $\\overline{d}, \\tilde{\\sigma}_d^2$ and $t$.\n",
    "---\n",
    "Let's calculate $\\overline{d}, \\tilde{\\sigma}_d^2$ and $t$. Recall:\n",
    "\n",
    "* Define our **performance measure** (pm): $$ pm_i(f) = \\begin{cases} 1, \\quad f \\text{ classified incorrectly } x_i \\\\ 0, \\quad \\text{otherwise} \\end{cases} $$ \n",
    "* The *average* performance measure of $f$: $$ \\overline{pm}(f) = \\frac{1}{n}\\sum_{i=1}^n pm_i(f) $$\n",
    "* Performance *difference* measures of two classifier $f_1$ and $f_2$: $$ d_i = pm_i(f_1) - pm_i(f_2) $$ $$ \\overline{d} = \\overline{pm}(f_1) - \\overline{pm}(f_2) $$\n",
    "* Define the t-statistics (as we do not know the mean nor the std): $$ t = \\frac{\\overline{d} - 0}{\\tilde{\\sigma}_d / \\sqrt{n}} $$\n",
    "* $\\tilde{\\sigma}_d^2 = \\frac{1}{n-1} \\sum_{i=1}^n (d_i -  \\overline{d})^2$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d_average:  0.14733000000000002\n",
      "empirical variance of d:  0.0030239951333333338\n",
      "empirical std of d:  0.054990864089713425\n",
      "t =  8.472286722254982\n"
     ]
    }
   ],
   "source": [
    "# let's calculate the wanted terms\n",
    "d_avg = d.mean()\n",
    "print(\"d_average: \", d_avg)\n",
    "\n",
    "var_d = (1 / (len(d) - 1)) * np.sum(np.square(d - d_avg))\n",
    "print(\"empirical variance of d: \", var_d)\n",
    "print(\"empirical std of d: \", np.sqrt(var_d))\n",
    "\n",
    "t_test = (d_avg - 0) / (np.sqrt(var_d) / np.sqrt(len(d)))\n",
    "print(\"t = \", t_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Section 4\n",
    "---\n",
    "#### Q: For the significane levels of $\\alpha=0.05$, do the two classifiers have similar performance?\n",
    "---\n",
    "* After calculating $t$, all that is left to be done is analyzing the distribution of $t$, which we know has the form of the **student-t distribution** with $n-1$ degrees of freedom (note that for $n \\to \\infty$ the student-t distribution approaches the Normal distribution). \n",
    "* In our case, $n=10$, since we averaged 10 RVs.\n",
    "\n",
    "* For two-tailed alternative hypothesis - we can read the value of $t_{n-1,1- \\frac{\\alpha}{2}}$ which set the t-test to have a certain size: $$ P(t > t_{n-1, 1- \\frac{\\alpha}{2}} ) = \\frac{\\alpha}{2} $$ Since we wish to have a test with size $\\alpha = 0.05$, we read the value: $$ t_{n-1, 1- \\frac{\\alpha}{2}} = t_{9, 1-0.025} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t value:  2.2621571627409915\n"
     ]
    }
   ],
   "source": [
    "alpha = 0.05\n",
    "t_value = t_dist.ppf(1 - alpha / 2, df=len(d) - 1)  # Percent point function (inverse of cdf — percentiles)\n",
    "print(\"t value: \", t_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So we got that: $$ t_{n-1,1- \\frac{\\alpha}{2}} = t_{9,1- 0.025} = 2.26 $$ And the test takes the following form: **reject** the null hypothesis $H_0$ if $$ |t| > t_{n-1, \\frac{\\alpha}{2}} = 2.26 $$\n",
    "\n",
    "Therfore, given the test size, $\\alpha = 0.05$, we reject $H_0$ and conclude that NB and c4.5 peform differently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Section 4 Continued\n",
    "---\n",
    "* For one-sided alternative hypothesis ($\\mu_1 > \\mu_2 \\rightarrow \\overline{d} > 0$) - we can read the value of $t_{n-1, 1-\\alpha}$ which set the t-test to have a certain size: $$ P(t > t_{n-1, 1-\\alpha} ) = \\alpha $$ Since we wish to have a test with size $\\alpha = 0.05$, we read the value: $$ t_{n-1, 1- \\alpha} = t_{9, 0.05} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t value:  1.8331129326536335\n"
     ]
    }
   ],
   "source": [
    "alpha = 0.05\n",
    "t_value = t_dist.ppf(1-alpha, df=len(d)-1)  # Percent point function (inverse of cdf — percentiles)\n",
    "print(\"t value: \", t_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So we got that: $$ t_{n-1, 1-\\alpha} = t_{9, 1-0.05} = 1.833 $$ And the test takes the following form: **reject** the null hypothesis $H_0$ if $$ t > t_{n-1, 1-\\alpha} = 1.833 $$\n",
    "\n",
    "Therfore, given the test size, $\\alpha = 0.05$, we reject $H_0$ and conclude that NB performs consistently *better* than c4.5 (since the expected error rate of c4.5 is higher)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/test-tube.png\" style=\"height:50px;display:inline\"> Pearson's Chi-Squared ($\\chi^2$) Test\n",
    "---\n",
    "* *Pearson’s chi-squared* test is a statistical test applied to sets of **categorical data** to evaluate how likely it is that any observed difference between the sets arose by chance.\n",
    "    * **Categorical data** - data consisting of categorical variables (a variable that can take on one of the limited possible values).\n",
    "        * For example - subway usage on specific days:\n",
    "      \n",
    "|Average Weekday| Average Saturday| Average Sunday|\n",
    "|---------------|-----------------|---------------|\n",
    "| 5,284,295     | 3,202,388       | 2,555,814     |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "* By using Pearson’s chi-squared test we can accept or reject the null hypothesis, $H_0$. Usually, a null hypothesis represented in the way of an expected set of data, and alternative hypothesis, $H_1$, as some newly observed data.\n",
    "* The Chi-Squared Statistic: $$ \\chi^2 = \\sum_{i=1}^n \\frac{(O[i] - E[i])^2}{E[i]} $$\n",
    "    * $O[i]$ - Observed data ($i^{th}$ category).\n",
    "    * $E[i]$ - Expected data (for the $i^{th}$ category).\n",
    "* We will use the **Chi-squared distribution function** with $n-1$ degrees of freedom (DOF), where $n$ is the number of categories.\n",
    "* For a **significance level** $\\alpha$, we reject the null hypothesis, $H_0$ if the chi-squared statistic is *above* the **critical value**.\n",
    "    * The **critical value** is calculated using the Chi-squared distribution function for a certain siginificance level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Example  - Dropout/Graduation Rates\n",
    "---\n",
    "Consider the following data about dropout/graduation rates (which will serve as our null hypothesis).\n",
    "\n",
    "|First Year Dropout| Second Year Dropout| Third Year Dropout|Fourth Year Dropout|Graduated|\n",
    "|------------------|--------------------|-------------------|-------------------|---------|\n",
    "|0.2               | 0.1                | 0.15              | 0.05              | 0.5     |\n",
    "\n",
    "Recently, a research was conducted and the observed data from a sample size of $N=2000$ was:\n",
    "\n",
    "|                |First Year Dropout| Second Year Dropout| Third Year Dropout|Fourth Year Dropout|Graduated|\n",
    "|----------------|------------------|--------------------|-------------------|-------------------|---------|\n",
    "|Number of People|543               | 145                | 210               | 110               | 992     |\n",
    "\n",
    "Assuming the old data was corerct ($H_0$), what would the expected data look like?\n",
    "\n",
    "|                |First Year Dropout| Second Year Dropout| Third Year Dropout|Fourth Year Dropout|Graduated| Total|\n",
    "|----------------|------------------|--------------------|-------------------|-------------------|---------|------|\n",
    "|Observed        |543               | 145                | 210               | 110               | 992     |2000  |\n",
    "|Expected        |$2000 \\cdot 0.2=400$|$2000 \\cdot 0.1=200$|$2000 \\cdot 0.15=300$|$2000 \\cdot 0.05=100$|$2000 \\cdot 0.5=1000$     |2000  |\n",
    "\n",
    "We will now perform the Chi-squared test for a significance level $\\alpha = 0.05$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chi-squared statistic value:  94.3115\n",
      "critical value:  9.487729036781154\n",
      "H_0 is rejected!\n"
     ]
    }
   ],
   "source": [
    "# chi-squared test\n",
    "num_categories = 5\n",
    "dof = num_categories - 1 # degrees of freedom\n",
    "alpha = 0.05\n",
    "observed = [543, 145, 210, 110, 992]\n",
    "expected = [400, 200, 300, 100, 1000]\n",
    "# compute chi-squared statistic\n",
    "chi_square_stat = np.sum((np.array(observed) - np.array(expected))**2 / np.array(expected))\n",
    "print(\"chi-squared statistic value: \", chi_square_stat)\n",
    "\n",
    "# compute critical value using the inverse of the CDF\n",
    "critical_value = chi2.ppf(q=1-alpha, df=dof)\n",
    "print(\"critical value: \", critical_value)\n",
    "\n",
    "if chi_square_stat > critical_value:\n",
    "    print(\"H_0 is rejected!\")\n",
    "else:\n",
    "    print(\"H_0 is accepted!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Chi-Squared Acceptance/Rejection Regions for alpha = 0.05')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfAAAAE/CAYAAAC5EpGHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxcVZ3//9enqtPZA2HrhBAWJYrBUcQMgsvYLjjAqNEZUZwRQZ1hmJFxHPWr6M+ZwR0X3BkRFAUHZVABo0bC2oawZiELIQnp7J09naXTa22f3x/3VnKpVHdXp6r7dnW9n49HPbrudu7nnq6qT51zT91r7o6IiIhUl0TcAYiIiMjAKYGLiIhUISVwERGRKqQELiIiUoWUwEVERKqQEriIiEgVUgKPiZldZ2b/28fylWbWOIQhDQkzazSzlrjjqGVm9g9mdn+1lDvYzKzdzF40hPszM/uZme0zs6eHYH8/N7MvV3pdiZ8S+CAys783s0XhB8R2M/uTmb2+lG3d/Wx3b+qj7I+Y2WozO2hmO83sj2Y2sWLBxyj8gFtvZs8Ng1iazOwf444DwMxea2aPh8/dzDrC19ZWM/u2mSVLKcfd73D3t5UZy+lhDHWVLLeXfTWaWS481oNmtsbMPlSp8t19gruvr1R5JXg9cCFwirufN4T7rQpmNtrMbjWzNjPbYWaf6Gf9vzezTeH74V4zOy6y7OdmlgpfO/lHSe+TaqAEPkjCF913ga8CDcCpwP8AsytQ9hvDct/v7hOBlwF3lVvuUcRR1/9aR+WvgJOAF5nZXw7SPqrRJcDcyPQr3X0C8EbgfcCHY4lqaGwLj3US8B/ALWb20phjOlqnARvdvWOgGw7ie244uQ6YQVBPbwI+bWYXFVvRzM4GfgxcTvA520nwORv1jfBLWv6RHbTIh5gS+CAws2OALwIfdfe73b3D3dPu/nt3/3+RVevN7PawVbHSzGZFythoZm/tZRd/CTzh7s8AuPted7/N3Q+G2x5vZnPCb7BPm9mXzGxBuOyIllO0lWlmLzazh82s1cz2mNkdZnZsQVyfMbPlQIeZ1ZnZyWb2WzPbbWYbzOxjkfXHht+C94Ut6lIS8hXA7wiS1RUFdXtc2P24LSzz3siy2Wa2NDzudfk3vZkdY2Y/DXtBtprZl/Pfws3sSjN7zMx+YGYHwl6Nt4TLvgK8Afhh+M39h+H875nZlnA/i83sDZEYrjOzu/r4v043s7vDumrNlxku+7CZrQqPa56ZnVZQL4UJHAB3bwYeA86JlNXfMS+IrHuWmT1gZnvD1u17I8vGmtkNYQvngJktMLOxwPxwlf1h3VxQpNzXmtnCcLuFZvbayLKm8HX5WFhP95vZCYXHVuRY3d3nAnuBV5R4DMeb2e/D/9fCsC6icbqZnRmpt9vD/88mM/u8mSWi9WZm3wr/RxvM7OJIOVda0HN0MFz2D4Xxm9lHgJ8AF4T19oVw/j+ZWXMY/xwzO7kgvo+a2VpgbbF6MbNfW9BaPWBm8y1IbMXWazSzFjP7nAXv741F4pxsQY/eQTN7ysxeHNm+19d+BX0Q+JK773P3VcAtwJW9rPsPwO/dfb67twP/CfytjZDeyH65ux4VfgAXARmgro91rgO6CT6Uk8DXgCcjyzcCb+1l2zcAXcAXgNcBowuW30nQIh8PvBzYCiwIl50OeDQ2oAn4x/D5mQTde6OBEwk+qL9bENdSYDowluBL4GLgv4B64EXAeuCvw/WvBx4Fjgu3eRZo6aNexgFtYb38HbAHqI8s/yPwf8BkYBTwxnD+ecCBMPYEMA04K1x2L8G39PEELfungX8Ol10Z/q/+IyzvfWE5xxXWTSSGDwDHA3XAJ4EdwJj+/q/h9DLgO2EsY4DXh8veBTQT9KbUAZ8HHo/sc2r4f7Rw2oEzw+dnAduB/4is398x518P44EtwIfC/Z4b1vnZ4fIbwzqYFsb/WoLXxukc+TqKlnscsI+gZVQHvD+cPj5Sr+uAlxC8jpqA63t5TTQSvmbC/+07gRzwqhKP4c7wMQ6YGa67IFJ+tC5vJ/jyODE8xueBj0SOLw38U1gX/wJsAyyMoQ14aeT/dXYvx3Nlwf7fHMZ7bli3PwDmF8T3QFinY3sp88NhzKMJev6WRpb9HPhypC4zwLfDdd8IdETi/jnBl6Pzwrq8A7izlNd+kZiuBfb39uhlm8nh8TZE5r0HWNHL+r8DPlMwrx14dcHx7CX4nPq7Sn7Wx/2IPYCR+CD4Vrijn3WuAx6MTM8EuiLTG+klgYfLLwZ+H74Z2sM3ZDJ8pAmTV7juVykxgRfZz7uAZwri+nBk+jXA5oJtPgv8LHy+Hrgosuwq+k7gHwB2hx8Qo8Pje3e4bCrBB/fkItv9GPhOkfkNQA+RDz6CZPJI+PxKwg/hyPKngcv7q5vI+vsIurP7/L8CF+SPrUgZfyJMFOF0gqA78LRw+iPATyPLnSBhdITPf0X4Ra7EY86/Ht4HPFqkLv87jKErf2wF6xR7HUXLvRx4umCbJ4ArI/X6+ciyfwXu66V+G8P/+/7wuLLAxyPL+zqG/PvhpZFlX6ZIAg/X7QFmRpb9M9AUOb7myLJx4bZTCBL4foIvnUWTbLF6Cqd/StDNm5+eEMZ8eiS+N/dVZkH5x4bbHBNO/5wjE/j4yPp3Af8ZWfcnkWWXAKtLee1X4kHwJd+JfCkg+FK+sZf1HwKuLpi3FWgMn5/L4S8clwAHgddVKt64H+pCHxytwAnW//mqHZHnncCYYtvYCwdgnArg7n9y93cQfCufTfCh8I8EreY6glZG3qZSAzezk8zszrDbtQ34X6CwazNa9mnAyWa2P/8APkeQRABOHmAsVwB3uXvG3XuAuzncjT4d2Ovu+4psN52gRVfoNIKW9fZIfD8maJXmbfXw3R6J8WR6YWafDLu6D4TlHcML66i3/+t0YJO7Z3qJ83uRGPcStOymhcuLdZ+fS/Bh/z6CL1LjB3DM0f2+puD/9w8ESekEgl6CYvXan5M58n+9KXI8cGQ9TeijvG3ufizBOfDvE7RaSzmGYu+H6POoEwh6kaJx9xqzu3eGTyd4cD77fcDVBPX+RzM7q4/jiXpBXXnQFdxasN/eYsbMkmZ2vQWnjdoIvmTnj6eYff7C8++Fr/de/y8lvPbL1R7+nRSZN4kg8fa2/qSCeYfWd/cl7t4afp7MJehR+NsKxhsrJfDB8QRBN+q7KlGYv3AAxuaCZTl3fwh4mKC7fDfBN+zpkdVOjTzPv3HHReZNiTz/GsE34Fe4+ySCFrEVhhR5vgXY4O7HRh4T3f2ScPn2PmJ5ATM7heCD+QPh+bwdBN1nl4TnR7cAx1nknHxBHC/uZX4PcEIkvknuHj1HOM3Mosd4KkGrvPBYCc/5fQZ4L0FPwLEEXe6FdVTMFuDUXr7YbSHo4o7W41h3f9zMRhF0dT5QuJEH7iJ4zf3XAI45ut8/F+x3grv/C0G3bjfF69WLzIvaRpBYo04laB0dtfBL3WeAvzCz/Purr2PIvx9OiRQzneL2ELR8o3GXHLO7z3P3Cwl6ilYTnLstxQvqyszGE7Qao/vtq77/nuBL/FsJEurp+aJ6WX9yuI+86Ou9VwN97Yfn2dt7exTbJvxyvh14ZWT2K4GVvYS1MrquBT8HHE1w6qPoLnqLtxopgQ8Cdz9A8GF6o5m9y8zGmdkoM7vYzL5RbvkWDNa6zMwmW+A8gg/4Jz0YYXk3cF2435lEBoK5+26CD4YPhN/cP8wLP6AnEnyr3W9m04DooLtingbaLBjYNjYs8+V2ePT4XcBnw1hPAf6tj7IuJ3jjvZRgQNY5BOdIWwhG3G8n6Gr+n7C8UWb2V+G2PwU+ZGZvMbOEmU0zs7PCbe4HbjCzSeGyF1swkj/vJOBjYXmXEpyHzrd2dxKc14/WT4awK9zM/osjWwB91dV24HozG29mY8zsdeGym8J6OhsODaa6NFz2BmC5u7f1Ufb1wFVmNqXEY877A/ASM7s8PP5RZvaXZvYyd88BtwLftmCgYtKCwWqjw+PPFdRN1Nyw3L+3YKDj+whOJ/yhxLrqlbungBs4/IWlr2MofD+cRTBIqli5WYLX61fMbKIFgwg/QdAL1SczazCzd4aJsYfgPVTqaOdfErx2zwnr9qvAU+6+scTtJ4b7bCX4Yv7VErb5gpnVh0n57cCvS9xPya99d/9qQePjBY8+9nM78PnwPX4WwZiDn/ey7h3AO8zsDWHdfxG42w8P6H2PmU0I3wNvI2iQzCnhWKuCEvggcfdvE7z5P0/wgt8CXEMwuKhc+whe1GsJzoP+L/BNd78jXH4NQbfXDoIX/s8Ktv8ngsTcCpwNPB5Z9gWCrtkDBAPG7u4rkPBD7x0EyXYDQSvmJwQtgXx5m8Jl9wO/6KO4K4D/cfcd0QdBcst/CbmcoJW0GtgFfDyM42mCQUzfCWP/M4dbNR8k6Bp9jqDufkPQSsp7iuBnK3uArwDvcffWcNn3gPdYMOr4+8A8gi8Rz4fH1U0f3Zu91NWZwGaCLybvC5fdA3wduDPsBn2WYJwD9DL6vKDsFeEx579w9XfM+e0OAm8DLiNohe0I4xgdrvIpYAWwkKBb/+tAIuw+/grwWNhtfX5Bua0EieGTBK+zTwNvd/c9fR3HANxK0JvxjhKO4RqC1+MOgtffrwgSXjH/RtBLtR5YQJBcby0hngTBsW4jqKc3EpzX71fYg/afwG8JvuC9ODyWUt1O8FrcSvD/frKf9XcQvCa2ESTAq919dQn7OerX/gD9N8Fpm00Er+lvuvt9+YVhC/4NAO6+kuC0xR0EnwcTeWG9/ztBvewHvgn8k/dxfY1qkx/RKiOYmV1JMBCrpIvI1JJqqBsLfn73HnevyIVtwl6XD7j7m/tdeQQys68DU9z9in5XHmEsuLrj/7r7Kf2tK8OfWuAiw5iZ1QO3Vyp5h84m6BGpCRb8RvwVkdNNHwHuiTsukXLVwlV9RKpWeL73+kqVZ8GFb2YAl/a37ggykaDb/GSCbtYbCH4/LFLV1IUuIiJShdSFLiIiUoWUwEVERKpQVZ0DP+GEE/z000+vWHkdHR2MHz++/xWlV6rDylA9lk91WD7VYfkqXYeLFy/e4+4nFltWVQn89NNPZ9GiRRUrr6mpicbGxoqVV4tUh5Wheiyf6rB8qsPyVboOzazXy0+rC11ERKQKKYGLiIhUISVwERGRKqQELiIiUoWUwEVERKqQEriIiEgVUgIXERGpQkrgIiIiVUgJXEREpAopgZdh9Y42DnSm4w5DRERqkBL4UdrZvoe/+cEjvOPmn7O/e3/c4YiISI1RAj8KmVyGN/3078lm69i44xjO/8nr6c50xx2WiIjUECXwo3DrM7eyc189AAnGs6N1Ml/88xdjjkpERGqJEvgAuTtfefQrkJ5Ojk6ytFGXeg3fffK77OncE3d4IiJSI5TAB+iZHc/Q2tlKfW4GqUQzncknGJd9De6juPHpG+MOT0REaoQS+AD95rnf0JPJUO9n0JNopjO5gATjsNRMfrjwh+Q8F3eIIiJSA5TAB+i3q36LZU/GqCdlzXQnlpHlAOOzwUC2+Zvmxx2iiIjUACXwATjQfYAN+zYwOncmAKlEM1iOzuTjjM2+hs6eDD9b+rOYoxQRkVqgBD4AT7Y8ydhRY6nPnUmODjK2HSDsRh/LmNy53Lv6XrK5bMyRiojISFdSAjezi8xsjZk1m9m1RZafZWZPmFmPmX0qMv+lZrY08mgzs4+Hy64zs62RZZdU7rAGx2NbHqMj1UF97kxSiXVgDkB3YgVZ9jMu+3rcnae3Ph1zpCIiMtL1m8DNLAncCFwMzATeb2YzC1bbC3wM+FZ0pruvcfdz3P0c4NVAJ3BPZJXv5Je7+9wyjmNI/Hnjn8nmCAewrT28wHJ0Jh9jbPY8ulJZ7l19b3xBiohITSilBX4e0Ozu6909BdwJzI6u4O673H0h0NeFwd8CrHP3TUcdbcxW7FrBKD81HMC27gXLOpILSDCG+sy53L367pgiFBGRWmHu3vcKZu8BLnL3fwynLwde4+7XFFn3OqDd3b9VZNmtwBJ3/2Fk3SuBNmAR8El331dku6uAqwAaGhpefeeddw7g8PrW3t7OhAkTSlo3m8uybOcynm2dxMMtU7n8rHUcO/rw95Wcw8+eO5Op4zv5m9O384qGV1CXqKtYrMPVQOpQeqd6LJ/qsHyqw/JVug7f9KY3LXb3WcWWlZJhrMi8vrN+YQFm9cA7gc9GZv8I+FJY1peAG4APH7Ej95uBmwFmzZrljY2NA9l1n5qamii1vD9v/DNfePoL1B38AOOZyJc3fvzQOfC8yf7PtB94G9dt+Bo/ecWNvPfs91Ys1uFqIHUovVM9lk91WD7VYfmGsg5L6UJvAaZHpk8Btg1wPxcTtL535me4+053z7p7DriFoKt+2Hpu93Oks+kjBrBFBaPRR5Ptmskf1/4xhihFRKRWlJLAFwIzzOyMsCV9GTBngPt5P/Cr6AwzmxqZfDfw7ADLHFLLdi6jK52m3k+nJ9FcdJ2exHNkaGV89g08uP7BIY5QRERqSb8J3N0zwDXAPGAVcJe7rzSzq83sagAzm2JmLcAngM+bWYuZTQqXjQMuBApHdn3DzFaY2XLgTcB/VOyoBsGyHcsiA9iKJ3DM6UwuYGzu1ezt6GRn+87i64mIiJSppFFW4U+85hbMuynyfAdB13qxbTuB44vMv3xAkcZs3b51jM69CgivwNaLzuSjTMrOZnzuAh7d/CjvmfmeoQpRRERqiK7EVoJUNsXerr3hFdjaD12BrZiexBoytptEzyweWv/QEEYpIiK1RAm8BJv2b2JM3Rjqc2fSk1hXfFx+3qFu9HN5cN1jQxajiIjUFiXwEqzft546G029n9Fn93leR/JRjFHsap1KZ7pzCCIUEZFaowRego37N5JLN2CM6n0AW0TKnidjO5mQewOLti0agghFRKTWKIGXoHlfM54+FYBU9BrovbHg0qp1mb/g4XVPDnJ0IiJSi5TAS7B692rqc2eSpZ2M7Shpm87koxh1zH22ZZCjExGRWqQEXoIN+zdQn3txeAW20rZJWTNp287WXVMGNzgREalJSuAl2H5wVziArYTu8zyD7sRikpkZbDmgVriIiFSWEng/srksnV3HljyALSqV2ECCcdy3ZuEgRSciIrVKCbwfOzt2MtZfCvR9BbZiUon1ADyydgAtdxERkRIogfejpa2F0T5jQAPY8tK2CSfLspbWQYpORERqlRJ4P7a2bSWZDS/gUuIAtjy3FGlrYe/BcbgP6BbqIiIifVIC78fm/dtIZqcPuPs8L53YQF32NDbu31jZwEREpKYpgffjue37ggFsAxmBHpGyDST9RB7duLjCkYmISC1TAu/H8zu6AQY8Aj0vP5DtoefXVCwmERERJfB+7Nw/miwHydjOo9o+ldgAwJItuyoZloiI1Dgl8H50dp14VAPY8nK2nwyt7D4wWgPZRESkYpTA+9CTyeLpqUc9gC0vndhAMnsaLW26IpuIiFSGEngfVu9ow6grO4GnEhuoy53Cwq3PVCgyERGpdUrgfXhywzbg6Aew5aVsPUYdD699rhJhiYiIKIH3ZcnmPeSs/agHsOXlR6I/tVFd6CIiUhlK4H1Yvb2DbHLDUQ9gy8vYdnJ0s22vqltERCpDGaUXPZksLXtzZJLryy/McqRtI7n0yezr2ld+eSIiUvOUwHuxdmc72ZzRbZW5k1gqsYFR/iKW7lhakfJERKS2lZTAzewiM1tjZs1mdm2R5WeZ2RNm1mNmnypYttHMVpjZUjNbFJl/nJk9YGZrw7+Tyz+cyjnQlQagJ7e7IuWlEutJ+Hjmr3+2IuWJiEht6zeBm1kSuBG4GJgJvN/MZhasthf4GPCtXop5k7uf4+6zIvOuBR5y9xnAQ+H0sNGVygLg1l2R8vID2R5dt6Ei5YmISG0rpQV+HtDs7uvdPQXcCcyOruDuu9x9IZAewL5nA7eFz28D3jWAbQddZzpI4Dl6KlJe2jbi5Fi3qzJfCEREpLaVksCnAVsi0y3hvFI5cL+ZLTazqyLzG9x9O0D496QBlDnouvMt8AolcLceMraNzq7jSWcH8j1HRETkSHUlrFPsR1QDuaj369x9m5mdBDxgZqvdfX6pG4dJ/yqAhoYGmpqaBrDrvrW3t/da3vJNQZK97szPMbYuW5H9/WnjJHZ1ncuDjzzI2LqxFSkzbn3VoZRO9Vg+1WH5VIflG8o6LCWBtwDTI9OnANtK3YG7bwv/7jKzewi65OcDO81sqrtvN7OpQNHbdbn7zcDNALNmzfLGxsZSd92vpqYmeitvVdM6WLWa/1p3LW6VaYVPSl/K5MwVbD22lX+c9YGKlBm3vupQSqd6LJ/qsHyqw/INZR2W0oW+EJhhZmeYWT1wGTCnlMLNbLyZTcw/B94G5IdhzwGuCJ9fAfxuIIEPtq5UBgAnVbEy87cWbWou79KsIiIi/bbA3T1jZtcA84AkcKu7rzSzq8PlN5nZFGARMAnImdnHCUasnwDcY2b5ff3S3e8Li74euMvMPgJsBi6t7KGVp62nmxzdYJW7BWg6HIm+rGVPxcoUEZHaVEoXOu4+F5hbMO+myPMdBF3rhdqAV/ZSZivwlpIjHWL7OjvBKtf6Bsiylyz72dM2pqLliohI7dGV2HrR1t1VsRHoh1h4QZfMdHa2l3eDFBERqW1K4L042NNT8RY45O8NfipLti+reNkiIlI7lMB70d6dqtjo86i0bcCoZ/66NRUvW0REaocSeC8605lgEFuF5S+p+sSGLf2sKSIi0jsl8F50pbJkvavi5aatBSfFxj26GpuIiBw9JfBedGdyg9ICx3KkbCM9XSfokqoiInLUlMB7kUozKOfA4fC9wdfs0XlwERE5OkrgvchkE5X/GVkondhAwiexYKPuDS4iIkdHCbwX2Vxy0BJ4yoKBbAt0b3ARETlKSuBFuDt4PW6Dc+/u/DXRl2/dOyjli4jIyKcEXkQqmwMS5AapBe7WRdq20do2Mm4pKiIiQ08JvIjuVA4YvEFsACnbgGWms69r36DtQ0RERi4l8CLaU8ElVAfrHDgEA9mSPoWFW5cP2j5ERGTkUgIvYnf7AWBwE3gqsR4jQdPa5wdtHyIiMnIpgRexuyNI4LlBGsQGh0eiP7Vx66DtQ0RERi4l8CJaO9qAwW2BZ20PWQ6yaU920PYhIiIjV13cAQxHrR3twOAmcAzSifVku44j5zkSpu9SIiJSOmWNIvZ1dQCDOwodgpHoo3Kn0dy6flD3IyIiI48SeBH7uzoBBudmJhHBQLbRPLx2xaDuR0RERh4l8CIOdAeJe1C70Dl8RbbHN+re4CIiMjBK4EUczCfwQe5CT9sWnDQrtupiLiIiMjBK4EW0hQl8sLvQsQxp20zrgdGDux8RERlxlMCLONiTwskB6UHfVyqxAc9MoyPVMej7EhGRkUMJvIiOVDo4/22Dv69UYj1JP47HNmkgm4iIlE4JvIjudBYf7O7zUMqCgWyP6JKqIiIyACUlcDO7yMzWmFmzmV1bZPlZZvaEmfWY2aci86eb2SNmtsrMVprZv0eWXWdmW81safi4pDKHVL5UevAHsOWlE8FvwBdv3jkk+xMRkZGh3yuxmVkSuBG4EGgBFprZHHd/LrLaXuBjwLsKNs8An3T3JWY2EVhsZg9Etv2Ou3+r7KOosHTWBu1e4IVy1kHGdrFh9+CfbxcRkZGjlBb4eUCzu6939xRwJzA7uoK773L3hRSM+nL37e6+JHx+EFgFTKtI5IMom0sO+m/Ao1K2nq6u43H3IduniIhUt1IS+DQgeqWRFo4iCZvZ6cCrgKcis68xs+VmdquZTR5omYMll6sbsi50CAey5U5m/V5d0EVEREpTys1Mio3FHlBT0cwmAL8FPu7ubeHsHwFfCsv6EnAD8OEi214FXAXQ0NBAU1PTQHbdp/b29qLlnVL/IsbWZfl/Lxqa3v3m/RP406YEcx9eyl+cWF3XRe+tDmVgVI/lUx2WT3VYvqGsw1ISeAswPTJ9CrCt1B2Y2SiC5H2Hu9+dn+/uOyPr3AL8odj27n4zcDPArFmzvLGxsdRd96upqYnC8jrTnWyddxsp28L8579WsX31pS7XwDR+ysLUdv6t8Z1Dss9KKVaHMnCqx/KpDsunOizfUNZhKV3oC4EZZnaGmdUDlwFzSinczAz4KbDK3b9dsGxqZPLdwLOlhTy42nraSDAat6H5GRlAxnaRo4Nnt+mSqiIiUpp+W+DunjGza4B5QBK41d1XmtnV4fKbzGwKsAiYBOTM7OPATOAVwOXACjNbGhb5OXefC3zDzM4h6ELfCPxzZQ/t6BzsOYgxZkgHsWFOKrGR1raxQ7dPERGpaqV0oRMm3LkF826KPN9B0LVeaAG9XM/M3S8vPcyhczB1EPPRQzqIDYKR6PXpt9KZ6mJcvRK5iIj0TVdiK9DW3QbUD/6NTAqkE+tJMJZH1umSqiIi0j8l8AKtnQcxEkPbhc7he4M3rW0e0v2KiEh1UgIvsK8ruCvYUHehp20zTpYlW3RJVRER6Z8SeIF9ne0AQ3Yzkzy3FGlrYfOe3JDuV0REqpMSeIH9XV0AQ3Yt9KhUYj3d3SfokqoiItIvJfACB7o7gaHvQgdI2waSfgIrd2wc8n2LiEh1UQIvcOgceEwtcIB5a1YO+b5FRKS6KIEXOBB2oQ/1OXA4PBL9qY1bh3zfIiJSXZTACxzsCVreuRi60HN2gAytrN5xcMj3LSIi1UUJvEB7T3BL8zi60AHSiQ3sPzghln2LiEj1UAIv0JXKAPEMYoPgPLhlp7K3s63/lUVEpGYpgRfoCfJ3LOfAAVK2AaOO+1Yv7X9lERGpWUrgBcIGeCy/A4fDI9EfXbchlv2LiEh1UAIvkMkaThbIxLN/206Obpa1tMayfxERqQ5K4AWyubpgAFvRm6AOAcuRto1s35eMKQAREakGSuAFcrlkbCPQ81KJDWRTU0hn07HGISIiw5cSeEQqm8J8NDmLZwDboTgS60kwgUc36IpsIiJSnBJ4REeqgwRjhkELPBjI9uCa1bHGISIiw5cSeER7qp0kY2NP4GnbiJNj0eYdsczSbI8AAB9dSURBVMYhIiLDlxJ4REe6A2N0bBdxyXPrIWPb2LBb58BFRKQ4JfCI9lQ7xmhyMV3EJSqVWE931/G6N7iIiBSlBB7RkerAPP4WOARXZEt6A8/uXBd3KCIiMgwpgUd0pMMEHvM5cDh8a9H7Vi2PORIRERmOlMAj2lPtQH1s10GPSocj0R/fsCXmSEREZDgqKYGb2UVmtsbMms3s2iLLzzKzJ8ysx8w+Vcq2ZnacmT1gZmvDv5PLP5zy5LvQ47gXeKEse8lygDU72uMORUREhqF+E7iZJYEbgYuBmcD7zWxmwWp7gY8B3xrAttcCD7n7DOChcDpW7alwFPow6ELHgoFsbe2TNJBNRESOUEoL/Dyg2d3Xu3sKuBOYHV3B3Xe5+0Kg8HdPfW07G7gtfH4b8K6jPIaKOdDdCTA8EjhBAk/mTmHT/pa4QxERkWGmlAQ+DYieiG0J55Wir20b3H07QPj3pBLLHDT7u8IEPgy60AHStgGjnvvWPBN3KCIiMszUlbBOsftyldqnW862QQFmVwFXATQ0NNDU1DSQzfvU3t7+gvJOa5sBwKUNs5l5fGPF9nO0Wrvq+eXzsG5lD03dTXGHU1RhHcrRUT2WT3VYPtVh+YayDktJ4C3A9Mj0KcC2Esvva9udZjbV3beb2VRgV7EC3P1m4GaAWbNmeWNjY4m77l9TUxPR8n74q2uAi7lz5y/obJ1fsf0cNU8wnf/jT7tX8c2PXBp3NEUV1qEcHdVj+VSH5VMdlm8o67CULvSFwAwzO8PM6oHLgDkllt/XtnOAK8LnVwC/Kz3swXGwO/j5mMd8N7JDLEcqsZb9B2MfoC8iIsNMvwnc3TPANcA8YBVwl7uvNLOrzexqADObYmYtwCeAz5tZi5lN6m3bsOjrgQvNbC1wYTgdq/buYAzecBnEBtCTWE0yeyrrWvV7cBEROayULnTcfS4wt2DeTZHnOwi6x0vaNpzfCrxlIMEOts50BoDcMEvgRh33rljCJxun97+BiIjUBF2JLaI7nQOGURc6QQIHmN+sFriIiBymBB6RChrgw6oLPWdtpG0ba7an4g5FRESGESXwiEMJfJj8DjyvJ7Gazs4Gcrlc3KGIiMgwoQQekckG1TEc7gce1ZNYTcKP5anNzXGHIiIiw4QSeEQmmwSGVxc6HD4PPufZFTFHIiIiw4USeEQul8TJgGXjDuUF0raRHN08taHotW5ERKQGKYGH3B3PjRoW9wI/QnhBl0279e8SEZGAMkKoK9NF0sYOq9+AR/UkVpNJTaGjR6PRRURECfyQznQnCcYMuxHoeT2JVRh1/OG55XGHIiIiw4ASeOhQAh+2LfA1ANy/am3MkYiIyHBQ0qVUa0GQwEcP2y70nB0gbdtZuiUTdygiIjIMqAUe6kx3Yj6G3DDtQofgPHjrgUm4D+iW6iIiMgIpgYc6051A/bDtQofw9+C5Y1i7uzXuUEREJGZK4KGgBT56WN3IpFAqvKDL3cuXxhyJiIjETQk81JXuChL4MG6Bp2wjOXpY0NwSdygiIhIzJfBQvgt9uA5iA8CypBLPs3aHBrKJiNQ6JfBQNXShQ3AevLv7RLpTSuIiIrVMCTzUke7ChvkgNggSuFHH/WtWxx2KiIjESAk8dLA7aHkP/wQeXNBl7nNK4CIitUwXcgm1hQl8OP8OHCBn+0nbDhZvGl53TBMRkaGlFnjoQFcXwPC8G1mBVGI1u/eP1wVdRERqmBJ46EBPPoEP7xY45C/ocixrd+2JOxQREYmJEniovTu4TedwvRtZVE9iFQC/WbY45khERCQuSuChjlQagFw1dKGHF3SZrwu6iIjULCXwUGf4u+pqaIEHF3RZS/NODWQTEalVJSVwM7vIzNaYWbOZXVtkuZnZ98Ply83s3HD+S81saeTRZmYfD5ddZ2ZbI8suqeyhDUx3OgdUxzlwCM6Dp3sa6OipjnhFRKSy+k3gZpYEbgQuBmYC7zezmQWrXQzMCB9XAT8CcPc17n6Ou58DvBroBO6JbPed/HJ3n1v20ZQhlQlGdFfDKHTIX9BlFPc8q/PgIiK1qJQW+HlAs7uvd/cUcCcwu2Cd2cDtHngSONbMphas8xZgnbtvKjvqQZBKGzD8fwee1xPemew+XdBFRKQmlZLApwFbItMt4byBrnMZ8KuCedeEXe63mtnkEmIZNOlckMCrpQs9Z/vJ2A6e2Xwg7lBERCQGpVyJzYrMK7yCSJ/rmFk98E7gs5HlPwK+FK73JeAG4MNH7NzsKoJueRoaGmhqaioh5NK0t7cfKu+1Exp54iB8/SVfIlHsaIaheZvGsq3j+IrWyUBF61COnuqxfKrD8qkOyzeUdVhKAm8BpkemTwG2DXCdi4El7r4zPyP63MxuAf5QbOfufjNwM8CsWbO8sbGxhJBL09TURL68yx65mtFcxKfXfqpi5Q+2iZm3c1z6apg+mcYXvzKWGKJ1KEdP9Vg+1WH5VIflG8o6LKULfSEww8zOCFvSlwFzCtaZA3wwHI1+PnDA3bdHlr+fgu7zgnPk7waeHXD0FZTzuuF9L/Ai8jc2uXvZkpgjERGRodZvC9zdM2Z2DTAPSAK3uvtKM7s6XH4TMBe4BGgmGGn+ofz2ZjYOuBD454Kiv2Fm5xB0oW8ssnxIea6uas5/56VsAzl6WLCusENERERGupLuRhb+xGtuwbybIs8d+Ggv23YCxxeZf/mAIh1E7g4+Crfq+AnZIZYhlVjDrn3H4u6YVcnJexERKZuuxAb0ZHtIMLbqWuAAXYnF1OVO59ENK+IORUREhpASONCV7iLBmKr5DXhUV3IRAHcseibmSEREZCgpgQNdmS7MR1fNVdii0raJDHt4rHl/3KGIiMgQUgIHujPdJBhTlV3oWNAKb2s7mZ5MJu5oRERkiCiBE3ShG6Or405kRXQlF5NgHL9equuii4jUCiVw8l3o9VVxL/BiuhNLcdLcvVTXRRcRqRVK4ARd6FRrFzrg1kVP4jmWb87FHYqIiAwRJXDCLnSvr9oudICuxCIyqZPYuEc3NxERqQVK4EB7qgtjVNW2wCE4Dw7w86efjjkSEREZCkrgwMHu4Nx3tV0LPSptm8nYLu5ftTXuUEREZAgogQMHe4LEXXWXUo2y4Kps2/ZMJJXRuXARkZFOCRw42B0m8CpugQN0JReCj+HB1RviDkVERAaZEjjQkUoDVPUgNoDuxHKcNL9avCzuUEREZJApgQMdPUECr+Zz4BCcAuhOPMvCDZ1xhyIiIoNMCRxo70kB1d+FDsF58O7uyWzZ2xF3KCIiMoiUwIl2oVfxILZQ/u5ktz+ty6qKiIxkSuAc7kIfCS3wjLWQsZ38aeWmuEMREZFBpAQOdKaDu3iNhAQe/JxsES17xtOTycYdjYiIDBIlcKArFSS6XJWPQs/rSi4CH82859bFHYqIiAwSJXCgJ7yNtlfp3cgK5X9OdseiZ+IORUREBokSOJDKOABOKuZIKsOth+7ECpZsHBk9CiIiciQlcCCVCZO3jZxLkHYlF5FOHc/zu1rjDkVERAaBEjiQySbIjZDu87yuRPBzslsefzzmSEREZDAogQPZbLLqL6NaKGPbSNt2Hli1Le5QRERkEJSUwM3sIjNbY2bNZnZtkeVmZt8Ply83s3Mjyzaa2QozW2pmiyLzjzOzB8xsbfh3cmUOaeCyucTI+AlZVPhzsn0HTqSjZ4Qdm4iI9J/AzSwJ3AhcDMwE3m9mMwtWuxiYET6uAn5UsPxN7n6Ou8+KzLsWeMjdZwAPhdOxyOXqRlwXOkBXcjHGaH761Py4QxERkQorpQV+HtDs7uvdPQXcCcwuWGc2cLsHngSONbOp/ZQ7G7gtfH4b8K4BxF1RudyoEdeFDtCTWEGOHn695Lm4QxERkQorJYFPA7ZEplvCeaWu48D9ZrbYzK6KrNPg7tsBwr8nDSTwivL6kdeFTvBzsp7ECjbumkAmm4k7HBERqaC6EtaxIvN8AOu8zt23mdlJwANmttrdS+7TDZP+VQANDQ00NTWVumm/2tvbaWpqYsqo6UyqT/PpM75VsbKHi2dbj+GRlincdPd9vPzECRUvP1+HUh7VY/lUh+VTHZZvKOuwlATeAkyPTJ8CFA5t7nUdd8//3WVm9xB0yc8HdprZVHffHna37yq2c3e/GbgZYNasWd7Y2FhCyKVpamqisbGRHfN+wqb0Gh55fuQlcPOxnMIv+OWW7Tx+6TUVLz9fh1Ie1WP5VIflUx2WbyjrsJQu9IXADDM7w8zqgcuAOQXrzAE+GI5GPx84ECbm8WY2EcDMxgNvA56NbHNF+PwK4HdlHstRcXfwenIjsAsdwK2LzuSjbN01lf1dI2+gnohIreo3gbt7BrgGmAesAu5y95VmdrWZXR2uNhdYDzQDtwD/Gs5vABaY2TLgaeCP7n5fuOx64EIzWwtcGE4PuZ5sDwlGj8hz4HkHk/MwxvDth5viDkVERCqklC503H0uQZKOzrsp8tyBjxbZbj3wyl7KbAXeMpBgB0NPpgdjDG4jt3WaSqwmZZu4e8k4vvg3cUcjIiKVUPNXYutIdWHUjegWOAbtdffT3nEiy1p2xx2NiIhUQM0n8P1dXQAj9hx4XkfyEZwM33zwsbhDERGRCqj5BH6gO0jgI7kLHSBnbXQmH+fx5zN0p7NxhyMiImWq+QTe1h0k7hHdhR5qT95PLjeWXy95Pu5QRESkTDWfwA92B4l7JF5KtVB3YhlZ28WPH10edygiIlImJfCwBT4Sb2ZyBHMOJufRsmcCm1o74o5GRETKUPMJvD2VBmqjCx2gve5BnBzffeSpuEMREZEy1HwC7+gJE3gNdKEDZK2V7sRi/rislUw2F3c4IiJylGo+gbf3pIDaaYEDHKybRyo9jrnPboo7FBEROUo1n8A708FtNmviHHioK7GQnO3ne48sjDsUERE5SkrgqSCB10oXOgCW5WDyAdbtGMOOA11xRyMiIkeh5hN4dyq4qEktdaEDtCcfAJJ886FH4w5FRESOQs0n8M50MJDLScUcydDKJLbRnVjOnKW7yeU87nBERGSAaj6B96RzwXXQrfaSWHvyftKpY7l3+cq4QxERkQFSAs/k8BoawBbVmXycHO1866HH4w5FREQGqOYTeHfaa2sAW4RbivbkI2zdfSIb9+o2oyIi1aTmE3hP2mtuAFtUe92fgDo+9ps5cYciIiIDUPMJPJUZ+fcC70s6sZmO5CMsW38czbv3xh2OiIiUqOYTeDprNduFnre/7hcA/Ntd98cciYiIlKrmE3gmm6jZQWx52cQe2up+x6otE1myWefCRUSqQc0n8Gw2UdPnwPMO1P2anLXx8V//Gffa+0mdiEi1UQLP1ZGr8S50ALdO9tf9ks27x3L/c1vjDkdERPpR8wk8l6ur+S70vIPJ+8gktvHZe5/WrUZFRIY5JfBcnbrQ8yzD3rqfsffgaG5/cm3c0YiISB9qOoG7O/iomh+FHtWVeIJUchXfmLeSjp5M3OGIiEgvSkrgZnaRma0xs2Yzu7bIcjOz74fLl5vZueH86Wb2iJmtMrOVZvbvkW2uM7OtZrY0fFxSucMqTdYBkjV1L/B+GbQmf0p3ajTfeuCZuKMREZFe9JvAzSwJ3AhcDMwE3m9mMwtWuxiYET6uAn4Uzs8An3T3lwHnAx8t2PY77n5O+Jhb3qEMXE9wJ1G1wAukkqvpSj7GbY9vZVebvtyIiAxHpbTAzwOa3X29u6eAO4HZBevMBm73wJPAsWY21d23u/sSAHc/CKwCplUw/rKkgia4zoEXsbfu52RzxqfvmR93KCIiUkQpCXwasCUy3cKRSbjfdczsdOBVwFOR2deEXe63mtnkEmOumEMtcCXwI2QS2zmYnEvTqh6e27Yv7nBERKRAXQnrWJF5hVf66HMdM5sA/Bb4uLu3hbN/BHwpXO9LwA3Ah4/YudlVBN3yNDQ00NTUVELIpdl/sBMwPjjtMl58zNsrVu5I0ZVJcvsq5xO/mM+1rzmm6Drt7e0V/Z/UKtVj+VSH5VMdlm8o67CUBN4CTI9MnwJsK3UdMxtFkLzvcPe78yu4+878czO7BfhDsZ27+83AzQCzZs3yxsbGEkIuzdp7HgK6+dm2m+neuaRi5Y4kkxJ/R2rfh9h1zGTe+6pXHrG8qamJSv5PapXqsXyqw/KpDss3lHVYShf6QmCGmZ1hZvXAZUDhvSfnAB8MR6OfDxxw9+1mZsBPgVXu/u3oBmY2NTL5buDZoz6Ko5RSF3q/2urmkLbtXPub59jZ1hV3OCIiEuo3gbt7BrgGmEcwCO0ud19pZleb2dXhanOB9UAzcAvwr+H81wGXA28u8nOxb5jZCjNbDrwJ+I+KHVWJevKD2EwjrXtlaXbXf5VsdjTvuukPpDK6QpuIyHBQShc64U+85hbMuyny3IGPFtluAcXPj+Pulw8o0kGQb4HX8v3AS5FObGDPqO9hez/NJ3/zGD+47A1xhyQiUvNq+kps3eH1vtWF3r/Ouvm01d3D75e28cvHdZlVEZG41XQCVxf6wOyr+xmv3biM6+5ZyfKW/XGHIyJS05TAUQu8ZJbjkwu+yYnt+7j6pvm0tqveRETiUtsJPOM4OZxU3KFUjdHZNn58z1do7c7y0R/PJ5srvCSAiIgMhZpO4KmcA6lehtlJb16+cx1fu+8HPLk7xV2rdPpBRCQONZ3AuzOuG5kcpb9d+QhXLvkD87bk+N0T6+IOR0Sk5tR0Ak9lUfd5Gf6/h27hJaM6+czdK1i5qTXucEREakptJ/AcYErgR2tULstHJ23l2K42rvz+QyxXEhcRGTI1ncB7sq6fkJXpmESWX9z5eUZ3dvDeGxdw3/KtcYckIlITajqBB1diUwu8XDNat3DP7Z/grF3r+Zc7nuHmh58nuDifiIgMlppP4BrEVhkndu7nzjuu5ZLmJ/jq/Wv53F1LSGd13XQRkcFS0wm8J+u6iEsFjcmk+MHdX+Nfn/4tv3pmBx++aQFt3em4wxIRGZFqOoGnc6YWeIUlcD79yM/4xn3f54lN+/i7Gx5iy97OuMMSERlxajqBp7K6E9lgee+y+7n9//6LnbsP8O5vPcAzm/bFHZKIyIhS8wnc0Sj0wfLazcu5+xefYty+Pbzvxkf5+t1LaO/JxB2WiMiIULMJ3N1J5yCnBD6oztzbwj23fYK3r57Pj57eTuOX7+OuhVvI6RrqIiJlqdkE3pPJ4RhZJfBBd3xXG9/+/Q3cc/snmL55LZ/+7XLe+Z1HeHrD3rhDExGpWjWbwLvTWQCydMUcSe141fbnufv2T/DdP36bPRtaeO+Pn+Cjtz9Nyz4NchMRGai6uAOIS2dwFRedAx9iBrzr2Yd525rHuOl1l/HjzGwefG4nV/3Vi/jwG2cweXx93CGKiFSFmk3gXWELXKPQ4zEu3cMnmm7jfYvn8vW3foQfkODH89dz4Vkn8Z4LzuCvZpxIMqH7vIqI9KZ2E3i+Ba7fgcdq2sHdfP+e6/mXE0/nrnMv5t6uN/DH1XuYMi7J3553OpfOms4ZJ4yPO0wRkWGndhN4Wl3ow8nLdm/kv+f9iGsfvIWHZ5zPr19xITd1pPmfpnX85bQJXHrBi/jrs6dwzNhRcYcqIjIs1G4CP3QOXC3w4WR0NsPFqxdw8eoF7JxwHHef89f8+mWNfHprO9f+ZhlnnziO88+awvkvOp6/POM4Jo1RQheR2lS7CTx/Dlxd6MNWQ/te/mXBr7h6wa9YcvJZzH/JeTw57Wxu2/ESbnl0FAmcl0+ZwPkvaeD8Fx3Hq089jmPGKaGLSG2o3QSuUehVw4BXb1vNq7etBqC7rp5nTn4pT545iyenvJSfb30pN88PEnfDmAQzTj6WM6dM4syTJjDjpAnMaJjIcRrdLiIjTEkJ3MwuAr4HJIGfuPv1BcstXH4J0Alc6e5L+trWzI4D/g84HdgIvNfdh+yC2YfOgasFXnXGZFJcsHkFF2xeAYQJfepLWXray2k+/hSaN0zlrhNOpXPUmEPbHD86wYtPnMDUEyYw5ZgxTJkUPBrC5ydNHE1dsmYviyAiVajfBG5mSeBG4EKgBVhoZnPc/bnIahcDM8LHa4AfAa/pZ9trgYfc/Xozuzac/kzlDq1vOgc+cozJpLhgywou2LLi0DwHtk88gbUnnsbaqS+m+bhTWDdpCosnnsCuCZNJJV/Y1W7ACWMSHD++nmMmjuWYcfUcM3YUx44bxTFjRx2anjimjnGjkoyrr2Pc6CTj6pOMG1XH2Pok9XX6AiAiQ6eUFvh5QLO7rwcwszuB2UA0gc8Gbnd3B540s2PNbCpB67q3bWcDjeH2twFNDGUCP/Q7cHWhj0QGnHxwDycf3MMb1y9+wTIH9o6dxI6Jx7PzmBPZMXkKOyaewM7xk9lXP5799ePYPHYi+8dO5MDo8XTVjS5pn6MMxtQZo5NGfTLB6FFBUq8flaR+VB31dcH0qKRRl0iQTBqjEkZdMsGunT08uH9FMD9hJBNGwoxkAhKWf354fsKC+WZgkemEAWYYkeXBLIzD6+d/YR9MB8vy04fqMJyIrhtMH7ludD0K1u9tjSOXH2kgVwJ4dmeG1ModA9ji8DFK4NldGdLP7Yw7jKrW3Jo9lNgGWykJfBqwJTLdQtDK7m+daf1s2+Du2wHcfbuZnVRs52Z2FXAVQENDA01NTSWE3L/Va1MYzjdfcj0JvYlL1nKtsyfDoU/f9mnTaLrhhniDKkMCODl8vFB38PBdpN3o8AQduSRdnqDnBQ974TRGxo10JkEmEz7H6HLjYPg8i5F1Iwtk3chhpBNJntm6ngyGAzk3coBz+K+U4JnF/a8jfVuyKO4Iqtop43K87PimIdlXKQm82CdH4a2kelunlG375O43AzcDzJo1yxsbGweyea9mvrqbs+Y9QM80fTAOROfudTRmpzMmbJU2wZB924xFvglLDpK58srK5SCVCh7pdPA3k4F0mqbzzqNx/vxgOv/IZg89PJslm3OyOSeXc9xz5LI5cu64B3fXy+WcnDs5gJyHXwSCN5zncsF6hG9A90PP3V84HTw3Dr2Fw2U4ePh28Re8i/3wF4xwQeGb/IjpEj4FBnq/usVXXMGrb7utSEG9lKS3/hEWffAKZt1epA6lZMs/9m80vvHtpXUxlamUBN4CTI9MnwJsK3Gd+j623WlmU8PW91Rg10ACL9dJE8dw9gkTabzgE0O52+p3QcF0UxO8731xRDKyNDXBT37S62IjeLPW7M9GSrCnqYmXf/LxuMOoanuamnj5p56IO4yqtqepaUiSN5R2N7KFwAwzO8PM6oHLgDkF68wBPmiB84EDYfd4X9vOAa4In18B/K7MYxEREakZ/X6hd/eMmV0DzCP4Kdit7r7SzK4Ol98EzCX4CVkzwc/IPtTXtmHR1wN3mdlHgM3ApRU9MhERkRGspB45d59LkKSj826KPHfgo6VuG85vBd4ykGBFREQkoB+uioiIVCElcBERkSqkBC4iIlKFlMBFRESqkBK4iIhIFVICFxERqUJK4CIiIlXIvJSLEg8TZrYb2FTBIk8A9lSwvFqkOqwM1WP5VIflUx2Wr9J1eJq7n1hsQVUl8Eozs0XuPivuOKqZ6rAyVI/lUx2WT3VYvqGsQ3Whi4iIVCElcBERkSpU6wn85rgDGAFUh5Wheiyf6rB8qsPyDVkd1vQ5cBERkWpV6y1wERGRqlSzCdzMLjKzNWbWbGbXxh1PNTCzW81sl5k9G5l3nJk9YGZrw7+T44xxuDOz6Wb2iJmtMrOVZvbv4XzVY4nMbIyZPW1my8I6/EI4X3U4QGaWNLNnzOwP4bTqcADMbKOZrTCzpWa2KJw3ZHVYkwnczJLAjcDFwEzg/WY2M96oqsLPgYsK5l0LPOTuM4CHwmnpXQb4pLu/DDgf+Gj42lM9lq4HeLO7vxI4B7jIzM5HdXg0/h1YFZlWHQ7cm9z9nMhPx4asDmsygQPnAc3uvt7dU8CdwOyYYxr23H0+sLdg9mzgtvD5bcC7hjSoKuPu2919Sfj8IMGH5zRUjyXzQHs4OSp8OKrDATGzU4C/AX4Sma06LN+Q1WGtJvBpwJbIdEs4Twauwd23Q5CcgJNijqdqmNnpwKuAp1A9DkjY9bsU2AU84O6qw4H7LvBpIBeZpzocGAfuN7PFZnZVOG/I6rBusAoe5qzIPA3HlyFjZhOA3wIfd/c2s2IvSemNu2eBc8zsWOAeM3t53DFVEzN7O7DL3RebWWPc8VSx17n7NjM7CXjAzFYP5c5rtQXeAkyPTJ8CbIsplmq308ymAoR/d8Ucz7BnZqMIkvcd7n53OFv1eBTcfT/QRDA2Q3VYutcB7zSzjQSnEN9sZv+L6nBA3H1b+HcXcA/B6dkhq8NaTeALgRlmdoaZ1QOXAXNijqlazQGuCJ9fAfwuxliGPQua2j8FVrn7tyOLVI8lMrMTw5Y3ZjYWeCuwGtVhydz9s+5+irufTvD597C7fwDVYcnMbLyZTcw/B94GPMsQ1mHNXsjFzC4hOAeUBG5196/EHNKwZ2a/AhoJ7razE/hv4F7gLuBUYDNwqbsXDnSTkJm9HngUWMHhc4+fIzgPrnosgZm9gmBwUJKgEXKXu3/RzI5HdThgYRf6p9z97arD0pnZiwha3RCcjv6lu39lKOuwZhO4iIhINavVLnQREZGqpgQuIiJShZTARUREqpASuIiISBVSAhcREalCSuAiIiJVSAlcRESkCimBi4iIVKH/H0QghpR+AjieAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot rejection/acceptance regions\n",
    "x = np.linspace(0, 50)\n",
    "fig = plt.figure(figsize=(8, 5))\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(x, chi2.pdf(x, df=dof))\n",
    "\n",
    "accept_region_xs = np.arange(0, critical_value, 0.001)\n",
    "ax.fill_between(accept_region_xs, chi2.pdf(accept_region_xs, df=dof), color='green')\n",
    "reject_region_xs = np.arange(critical_value, 50, 0.001)\n",
    "ax.fill_between(reject_region_xs, chi2.pdf(reject_region_xs, df=dof), color='red')\n",
    "ax.grid()\n",
    "ax.set_title(\"Chi-Squared Acceptance/Rejection Regions for alpha = 0.05\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/50/000000/flex-biceps.png\" style=\"height:50px;display:inline\"> Uniformly Most Powerful Test (UMP)\n",
    "---\n",
    "* A Uniformly Most Powerful (UMP) test has the most statistical power from the set of all possible alternate hypotheses of the same size $\\alpha$.\n",
    "* The UMP doesn’t always exist, especially when the test has nuisance variables (variables that are irrelevant to your study but that have to be be accounted for).\n",
    "* However, if the UMP does exist, you can use the Neyman-Pearson lemma (NPL) to find it.\n",
    "* A UMP test is usually defined in terms of a uniformly most powerful rejection region (UMPCR) (also called a “critical region”).\n",
    "    * A region $C$ of size $\\alpha$ is the UMPCR for testing a simple null hypothesis against a set of alternate hypotheses if it is the “best” critical region.\n",
    "    * The “best” critical region is one that minimizes the probability of making a Type I or a Type II error.\n",
    "        *  In other words, the UMPCR is the region that gives the smallest chance of making a Type I or II error. It is also the region that gives a UMP test the largest (or equally largest) power function.\n",
    "* Formal definition: a test defined by a critical region $C$ of size $\\alpha$ is a uniformly most powerful (UMP) test if it is a most powerful test against each simple alternative in the alternative hypothesis $H_A$. The critical region $C$ is called a uniformly most powerful critical region of size $\\alpha$.\n",
    "\n",
    "* UMP and the **Neyman-Pearson Lemma**:\n",
    "    * The Neyman-Pearson lemma can tell you the best hypothesis test if you have a simple null hypothesis and a simple alternative hypothesis. \n",
    "        * If you have multiple hypotheses (also called a composite hypothesis), the NPL can be extended to all individual alternate hypotheses. \n",
    "        * Composite hypotheses have multiple options for solutions. For example, $\\sigma^2 > 8$ is a composite hypothesis because it doesn’t specify a value for $\\sigma^2$\n",
    "    * The basic idea is that you test each simple hypothesis in turn to see if it is the UMP out of all possibilities.\n",
    "    * Formally:\n",
    "        * Let $C$ be a class of tests for testing $H_0$: $\\theta \\in \\Theta_0$ versus $H_1$: $\\theta \\in \\Theta^c_1$. A test in class $C$, with power function $\\beta(\\theta)$, is a uniformly most powerful (UMP) class $C$ test if $\\beta(\\theta) \\geq \\beta'(\\theta)$ for every $\\theta \\in \\Theta^c_0$ and every $\\beta'(\\theta)$ that is a power function of a test in class $C$.\n",
    "        * In more simple terms, this is really just telling you that the UMP test is the one with the biggest power function (out of all tests of the same size $\\alpha$). $\\Theta_0$ is the set of all possible values for $\\theta$ under the null hypothesis.\n",
    "        * The same statement can be rewritten using the **likelihood ratio test**. Let’s say you had two simple hypotheses $H_0$: $\\theta = \\theta_0$ and $H_1$:$\\theta = \\theta_1$. In order to find the most powerful test at a certain $\\alpha$ level (with threshold $\\eta$), you would look for the likelihood-ratio test which rejects the null hypothesis in favor of the alternate hypothesis when: $$ \\Lambda(x) = \\frac{L(x|\\theta_0)}{L(x|\\theta_1)} \\leq \\eta $$ Where $$P(\\Lambda(X) \\leq \\eta|H_0) = \\alpha $$\n",
    "            * $L$ is the likelihood function.\n",
    "        \n",
    "* <a href=\"https://online.stat.psu.edu/stat415/lesson/26/26.2\"> Exercise Example </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/100/000000/ask-question.png\" style=\"height:50px;display:inline\"> Which Statistical Test To Use???\n",
    "---\n",
    "<img src=\"./assets/tut_04_how_to_choose.png\" style=\"height:500px\">\n",
    "\n",
    "* Note that for $\\chi^2$ the DOF are for the general case where the data is a table with $r$ rows and $c$ columns. When the data is a single row, as in the above exercise, the DOF is just $n-1$ where $n$ is the number of categories.\n",
    "* <a href=\"http://isoconsultantpune.com/hypothesis-testing/\">Image Source</a>\n",
    "* Video: https://www.youtube.com/watch?v=rulIUAN0U3w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "\n",
    "* t-Test - <a href=\"https://www.youtube.com/watch?v=AGh66ZPpOSQ\">T-Tests: A Matched Pair Made in Heaven</a>\n",
    "* $\\chi^2$-Test - <a href=\"https://www.youtube.com/watch?v=WXPBoFDqNVk\">Chi-squared Test</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/50/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "* Examples, exercises and definitions from <a href=\"https://probabilitycourse.com/\">Introduction to Probability, Statistics and Random Processes</a> - https://probabilitycourse.com\n",
    "* <a href=\"https://medium.com/@geekrodion/statistics-pearsons-chi-squared-test-95fe36d74c1c\">Pearson's chi-squared test</a> by Rodion Chachura\n",
    "* <a href=\"https://www.statisticshowto.datasciencecentral.com/ump-uniformly-most-powerful/\">Uniformly Most Powerful (UMP) Test </a> by Statistics How To\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com\n",
    "* Datasets from <a href=\"https://www.kaggle.com/\">Kaggle</a> - https://www.kaggle.com/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}