04- Bayesian Models/Lab/Probability and Bayesian Theory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "790ce472",
   "metadata": {},
   "source": [
    "## Machine Learning for Neuroscience, <br>Department of Brain Sciences, Faculty of Medicine, <br> Imperial College London\n",
    "### Contributors: Francesca Palermo, Nan Fletcher-Lloyd, Alex Capstick, Yu Chen\n",
    "**Winter 2022**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9246f7fe",
   "metadata": {},
   "source": [
    "# Probability and Bayesian Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b161ecd",
   "metadata": {},
   "source": [
    "This tutorial will focus on probability and Bayesian theory using the machine learning library for Python scikit-learn (https://scikit-learn.org/stable)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb7b588",
   "metadata": {},
   "source": [
    "## Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa77cbf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn as sk\n",
    "from sklearn import datasets\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5b10c736",
   "metadata": {},
   "source": [
    "## Probability Theory"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0cebf2f0",
   "metadata": {},
   "source": [
    "In this section, we will introduce you to hypothesis testing. \n",
    "\n",
    "Hypothesis testing is a statistical method that is used to make decisions using experimental data. \n",
    "\n",
    "A hypothesis is basically an assumption made about a population parameter and a test evaluates two mutually exclusive statements about a population to determine which statement is best supported by the data. \n",
    "\n",
    "These statements are the null and alternative hypothesis.\n",
    "\n",
    "The null hypothesis suggests that no statistical significance exists in a set of given observations.\n",
    "\n",
    "The alternative hypothesis is contrary to the null hypothesis. It is usually taken to be that the observations are the result of a real effect.\n",
    "\n",
    "Read more about the key terms of hypothesis testing and the different types and when to use them here: https://towardsdatascience.com/hypothesis-testing-in-machine-learning-using-python-a0dc89e169ce"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b4123af7",
   "metadata": {},
   "source": [
    "Here, we will demonstrate how, with enough data, statistics can enable us to calculate probabilities using real-world observations. \n",
    "\n",
    "Here, probability provides the theory while statistics provides the tools to test that theory using data.\n",
    "\n",
    "In the end, descriptive statistics such as the mean and standard deviation of the data become proxies for theoretical. This is because real-world probabilities are often quite difficult to calculate. As such, we rely on statistics and data.\n",
    "\n",
    "With more and more data, we can become more confident that what we calculate represents the true probability of these events occurring."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36eb0b85",
   "metadata": {},
   "source": [
    "So let's start with a question. For this, we are going to download the wine dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fed4d1e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.load_wine(as_frame=True)\n",
    "features = data.data # this derives features as a dataframe (13 features by 178 instances)\n",
    "features.columns = ['Alcohol','Malic Acid','Ash','Alcalinity of Ash','Magnesium','Total Phenols','Flavanoids', 'Nonflavanoid Phenols','Proanthocyanins','Color Intensity','Hue','OD280/OD315 of Diluted Wines','Proline']\n",
    "labels = data.target # this derives labels as a dataframe (178 instances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74b238c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = pd.concat([features, labels], axis=1,)\n",
    "wine"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e481dc32",
   "metadata": {},
   "source": [
    "Now, a quick check will remind us that there are three classes of wine in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a423cd36",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels.unique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "84794a5f",
   "metadata": {},
   "source": [
    "For this question, we will compare two of the three different types of wine across one feature variable, removing the need for scaling. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "af946bab",
   "metadata": {},
   "source": [
    "So, let's pose our question. Is the level of alcohol different between classes 0 and 1?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7a35af0a",
   "metadata": {},
   "source": [
    "First, we select for these classes and the column of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e084cc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = wine.iloc[:, [0,13]] # selects for the feature variable\n",
    "question"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0a5cea80",
   "metadata": {},
   "source": [
    "Next, we need to group our 'question' dataframe by class. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "61a10a47",
   "metadata": {},
   "source": [
    "A quick check of the data shows that the target is an integer. To split out dataframe by group, we need this to be a str."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c97dab55",
   "metadata": {},
   "outputs": [],
   "source": [
    "question.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11bddc85",
   "metadata": {},
   "outputs": [],
   "source": [
    "question['target'] = question['target'].astype(str) #converts the target column to str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c97dab55",
   "metadata": {},
   "outputs": [],
   "source": [
    "question.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "556a4f0b",
   "metadata": {},
   "source": [
    "Now, we can groupby target and get each group out as a separate dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc7db780",
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = question.groupby('target')\n",
    "class_0 = grouped.get_group('0')\n",
    "class_1 = grouped.get_group('1')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c486846d",
   "metadata": {},
   "source": [
    "A quick check let's us know this has worked as planned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c974b2f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c974b2f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_1"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dc3da062",
   "metadata": {},
   "source": [
    "We now want to use these values of the level of alcohol of the two different classes of wine to compare groups but these scores fall in a range. So how do we compare groups of values between types of wines and know with some degree of certainty that one is different from the other?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "98069f05",
   "metadata": {},
   "source": [
    "For this, we first need to understand the distribution of these data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a4b682ea",
   "metadata": {},
   "source": [
    "So let's visualise each group of values as histograms.\n",
    "\n",
    "Learn more about using histograms here: https://seaborn.pydata.org/generated/seaborn.histplot.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05a86917",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([class_0, class_1])\n",
    "ax = sns.histplot(data=df, x=\"Alcohol\", hue=\"target\", element=\"step\", bins=25, palette='PuRd')\n",
    "sns.move_legend(ax, title='Class',loc='lower center', bbox_to_anchor=(0.5, -0.4), ncol=2)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea4ad97a",
   "metadata": {},
   "source": [
    "Here we see that the data in each class is approximately normally distributed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fbccf80e",
   "metadata": {},
   "source": [
    "In probability, the normal distribution is a particular distribution of the probability across all of the data. The x-axis takes on the values of the data and the y-axis is the probability associated with each datapoint, from 0 to 1.\n",
    "\n",
    "The high point of the distribution represents the datapoint with the highest probability of occurring. As you move further away from this point on either side, the probability of those points decreases, forming the familiar bell-shape curve. The high point in a statistical context actually represents the mean, and as you move farther from the mean, frequency rapidly decreases."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dfbf3645",
   "metadata": {},
   "source": [
    "If any two distributions overlap significantly, we might assume they actually come from the same distribution and that there is no real difference in the means of those dsitributions. If there is no overlap, it is safe to assume that the distributions aren't the same. But, as you will see in the plot above, the difficulty is when there is some overlap."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4e1b4459",
   "metadata": {},
   "source": [
    "The normal distribution is significant to probability and statistics due to two factors: the Central Limit Theorem and the Three Sigma Rule.\n",
    "\n",
    "The key tenet of the Central Limit Theorem is that with more data, the closer the average of these data to the true probability. The Central Limit Theorem dictates that if we collect many data, the distribution of these data will look like a normal distribution and the high point of this distribution will align with the true value that the estimates should take on, i.e. the average of these many data will approach the true mean.\n",
    "\n",
    "The Three Sigma Rule tells us how much the data will be spread out aroud this mean or how many of the data falls within a certain distance of the mean. The standard deviation is the average distance a singular measurement in the dataset is from the mean. The Three Sigma Rule dictates that given a normal distribution, 68% of the measurements will fall between one standard deviation of the mean, 95% within two, and 99.7% within three. This rule is also a statement of rarity of extreme values."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1156317e",
   "metadata": {},
   "source": [
    "By taking advantage of the Three Sigma Rule, we can prescribe a value to how likely it is that the the level of alcohol is different within each class of wine."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d1643dc0",
   "metadata": {},
   "source": [
    "Now, the next thing we want to do is turn our question into a null and alternative hypothesis, as this will allows us to carry out hypothesis testing.\n",
    "\n",
    "Question: Is the level of alcohol different between classes 0 and 1?\n",
    "\n",
    "So, our null hypothesis is that the level of alcohol is no different between classes 0 and 1 and our alternative hypothesis is that the level of alcohol is different between classes 0 and 1."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3cb8a83d",
   "metadata": {},
   "source": [
    "If you read through the article from earlier (https://towardsdatascience.com/hypothesis-testing-in-machine-learning-using-python-a0dc89e169ce) you will know that there are several different types of statistical tests that can be conducted to test a hypothesis.\n",
    "\n",
    "Here, we want to test whether the means of two independent groups are equal or not. As such, we know we will be conducting a two sample, two-tailed test.\n",
    "\n",
    "We are assuming that our data is normally distributed but we also know that the sample size of each group is greater than 30 so this does not matter quite so much. It is also possible to calculate the mean and standard deviation of both sample groups. Given this, we are going to run a z-test."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e299fd95",
   "metadata": {},
   "source": [
    "First, let's calculate the mean and standard deviation of the level of alcohol of each class of wine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0384a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import mean\n",
    "from numpy import std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1470a41",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_0_avg = np.mean(class_0.Alcohol) # calculates variable mean\n",
    "class_0_std = np.std(class_0.Alcohol) # calculates variable standard deviation\n",
    "class_1_avg = np.mean(class_1.Alcohol)\n",
    "class_1_std = np.std(class_1.Alcohol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9cbccec",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Class 0: \", class_0_avg, class_0_std)\n",
    "print(\"Class 1: \", class_1_avg, class_1_std)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "36be67a6",
   "metadata": {},
   "source": [
    "These values look very similar! But let's calculate the z-test statistic before making any final decisions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7688f42c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.stats.weightstats import ztest"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "89278020",
   "metadata": {},
   "source": [
    "*N.B. you can learn more about using the statsmodels z-test here: https://www.statsmodels.org/dev/generated/statsmodels.stats.weightstats.ztest.html*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e4668ffc",
   "metadata": {},
   "source": [
    "This method requires the dataframes to be in arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cc1e690",
   "metadata": {},
   "outputs": [],
   "source": [
    "class_0 = class_0['Alcohol'].values\n",
    "class_1 = class_1['Alcohol'].values"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6a863c85",
   "metadata": {},
   "source": [
    "Finally, we can calculate the test-statistic and p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5ccd584",
   "metadata": {},
   "outputs": [],
   "source": [
    "ztest(class_0, class_1, value=0, alternative='two-sided') "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a3d417e",
   "metadata": {},
   "source": [
    "The p-value tells us whether to accept or reject the null hypothesis. 100% accuracy is not possible for accepting or rejecting a hypothesis; therefore, we select a level of significance that is usually 5%. As such, if the p-value > 0.05, then the null hypothesis is accepted, but if the p-value < 0.05, then the null hypothesis is rejected."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1e8d36fc",
   "metadata": {},
   "source": [
    "In this case, we reject the null hypothesis. The level of alcohol is different between classes 0 and 1."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "db309354",
   "metadata": {},
   "source": [
    "And that's it! You've now learnt the basics of hypothesis testing."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5b10c736",
   "metadata": {},
   "source": [
    "## Bayesian Theory"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0fea29ed",
   "metadata": {},
   "source": [
    "The second part of this tutorial will focus on Bayesian theory."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "98d554d1",
   "metadata": {},
   "source": [
    "Now, as you will have learnt in your lectures, Naive Bayes methods are a set of supervised learning methods (classifiers) that assume that the presence of a particular feature in a class is unrelated to the presence of any other feature.\n",
    "\n",
    "There are several types of Naive Bayes models (learn more here: https://scikit-learn.org/stable/modules/naive_bayes.html#gaussian-naive-bayes) but for this exercise, we will focus on Gaussian Naive Bayes classifiers.\n",
    "\n",
    "Learn more about how to use a Gaussian Naive Bayes classifier here: https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html#sklearn.naive_bayes.GaussianNB"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ec4e67ba",
   "metadata": {},
   "source": [
    "For this, we will create our own dataset of 10 features across 500 instances, categorised into two classes. Class 0 represents a control group and Class 1 the class of interest. As in real world medical problems, there are fewer isntances of the class of interest.\n",
    "\n",
    "Learn more about making your own classification datasets here: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10f0b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "features, labels = datasets.make_classification(n_samples=500, n_features=10, n_informative=8, n_classes=2, weights=np.array([0.65, 0.35]), flip_y=0.01, class_sep=2.0, shuffle=True, random_state=42)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "de601e90",
   "metadata": {},
   "source": [
    "Now, let's take a quick look at the features and labels that were generated, in dataframe form for ease."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60c63302",
   "metadata": {},
   "outputs": [],
   "source": [
    "features = pd.DataFrame(features)\n",
    "labels = pd.DataFrame(labels)\n",
    "df = pd.concat([features, labels], axis=1)\n",
    "df.columns = ['F1','F2','F3','F4','F5','F6','F7','F8','F9','F10','Class']\n",
    "df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e9a03dfc",
   "metadata": {},
   "source": [
    "Now, as we are working with Naive Bayes classifer, we do not need to do any feature scaling, so let's get straight into building our model. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9343433f",
   "metadata": {},
   "source": [
    "First, we must split our data into training and testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc6cc873",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_train, x_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, shuffle=True, random_state = 42)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c21e3f0e",
   "metadata": {},
   "source": [
    "Next, we need to import and build our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9beeafef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "GNB = GaussianNB(var_smoothing=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d118a892",
   "metadata": {},
   "outputs": [],
   "source": [
    "GNB.fit(x_train, y_train) # fits the model on the training data\n",
    "y_pred = GNB.predict(x_test) # predicts labels on the test data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "12d324a1",
   "metadata": {},
   "source": [
    "Now, we want to evaluate the performance of the classification model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a7164ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5d325a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 = f1_score(y_test, y_pred)\n",
    "recall = recall_score(y_test, y_pred)\n",
    "precision = precision_score(y_test, y_pred)\n",
    "\n",
    "f1_avg = mean(f1_score(y_test, y_pred, average=None))\n",
    "recall_avg = mean(recall_score(y_test, y_pred, average=None))\n",
    "precision_avg = mean(precision_score(y_test, y_pred, average=None))\n",
    "\n",
    "f1_sd = std(f1_score(y_test, y_pred, average=None))\n",
    "recall_sd = std(recall_score(y_test, y_pred, average=None))\n",
    "precision_sd = std(precision_score(y_test, y_pred, average=None))\n",
    "\n",
    "print('\\nf1:\\t\\t',f1)\n",
    "print('recall\\t\\t',recall)\n",
    "print('precision\\t',precision)\n",
    "\n",
    "print('\\nf1_avg:\\t\\t',f1_avg)\n",
    "print('recall_avg\\t',recall_avg)\n",
    "print('precision_avg\\t',precision_avg)\n",
    "\n",
    "print('\\nf1_sd:\\t\\t',f1_sd)\n",
    "print('recall_sd\\t',recall_sd)\n",
    "print('precision_sd\\t',precision_sd)\n",
    "\n",
    "print('\\n',classification_report(y_test, y_pred))\n",
    "print(roc_auc_score(y_test, y_pred))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "40c1815b",
   "metadata": {},
   "source": [
    "Now, these scores are not very good, particularly when it comes to identifying the class of interest (Class 1). So how can we improve model performance?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e4d622d0",
   "metadata": {},
   "source": [
    "Well, one method is to tune the hyperparameters of the model (read more about this here: https://medium.com/analytics-vidhya/how-to-improve-naive-bayes-9fa698e14cba)\n",
    "\n",
    "For this, we first need to import the sklearn GridSearchCV function. This functions runs through all the different parameters fed into the parameter grid and produces the best combination of parameters based on a chosen scoring metric.\n",
    "\n",
    "Learn more about how to use this function here: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cd658be",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "603c043d",
   "metadata": {},
   "source": [
    "Next, we need to set the range of all the parameters we will feed into the parameter grid. For a Gaussian Naive Bayes classifier, this is just var_smoothing, which represents a stability calculation to widen (or smooth) the curve and therefore account for more samples that are further away from the distribution mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bd54c20",
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid_nb = {'var_smoothing': np.logspace(0,-9, num=1000)} # np.logspace returns numbers spaced evenly on a log scale, starts from 0, ends at -9, and generates 1000 samples"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3ea1a849",
   "metadata": {},
   "source": [
    "Now, we build the GridSearchCV, using the model and parameter grid. We then fit this searching tool to the training data using a 10-fold cross-validation to find an optimal combination of hyperparameters that minimizes a predefined loss function to give better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1e9b352",
   "metadata": {},
   "outputs": [],
   "source": [
    "GNB_Grid = GridSearchCV(estimator=GaussianNB(var_smoothing=0.5), param_grid=param_grid_nb, verbose=1, cv=10, n_jobs=-1)\n",
    "GNB_Grid.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4887cea",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(GNB_Grid.best_estimator_)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cdcc4ab1",
   "metadata": {},
   "source": [
    "Now we want to re-evaluate the model on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ff0e5a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "grid_pred = GNB_Grid.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5d325a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 = f1_score(y_test, grid_pred)\n",
    "recall = recall_score(y_test, grid_pred)\n",
    "precision = precision_score(y_test, grid_pred)\n",
    "\n",
    "f1_avg = mean(f1_score(y_test, grid_pred, average=None))\n",
    "recall_avg = mean(recall_score(y_test, grid_pred, average=None))\n",
    "precision_avg = mean(precision_score(y_test, grid_pred, average=None))\n",
    "\n",
    "f1_sd = std(f1_score(y_test, grid_pred, average=None))\n",
    "recall_sd = std(recall_score(y_test, grid_pred, average=None))\n",
    "precision_sd = std(precision_score(y_test, grid_pred, average=None))\n",
    "\n",
    "print('\\nf1:\\t\\t',f1)\n",
    "print('recall\\t\\t',recall)\n",
    "print('precision\\t',precision)\n",
    "\n",
    "print('\\nf1_avg:\\t\\t',f1_avg)\n",
    "print('recall_avg\\t',recall_avg)\n",
    "print('precision_avg\\t',precision_avg)\n",
    "\n",
    "print('\\nf1_sd:\\t\\t',f1_sd)\n",
    "print('recall_sd\\t',recall_sd)\n",
    "print('precision_sd\\t',precision_sd)\n",
    "\n",
    "print('\\n',classification_report(y_test, grid_pred))\n",
    "print(roc_auc_score(y_test, grid_pred))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "94211839",
   "metadata": {},
   "source": [
    "Now, these scores are much better!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ed890df2",
   "metadata": {},
   "source": [
    "And there you have it! You've now built and tuned your first Bayesian classifer."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd194e07",
   "metadata": {},
   "source": [
    "Now you've finished this tutorial, follow the instructions and complete the assessment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8 (default, Apr 13 2021, 12:59:45) \n[Clang 10.0.0 ]"
  },
  "vscode": {
   "interpreter": {
    "hash": "730e964555fb2542a911b5002c3c4a5ea6b8ea7e74d00811d465953d33b870ef"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}