worksheet_classification2/worksheet_classification2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "655a4475483861185b311a37600fa826",
     "grade": false,
     "grade_id": "cell-83e402ae256bacca",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "# Worksheet 7 - Classification (Part II)\n",
    "\n",
    "### Lecture and Tutorial Learning Goals:\n",
    "\n",
    "After completing this week's lecture and tutorial work, you will be able to:\n",
    "\n",
    "* Describe what a test data set is and how it is used in classification.\n",
    "* Using R, evaluate classification accuracy using a test data set and appropriate metrics.\n",
    "* Using R, execute cross-validation in R to choose the number of neighbours.\n",
    "* Identify when it is necessary to scale variables before classification and do this using R\n",
    "* In a dataset with > 2 attributes, perform k-nearest neighbour classification in R using the `tidymodels` package to predict the class of a test dataset.\n",
    "* Describe advantages and disadvantages of the k-nearest neighbour classification algorithm.\n",
    "\n",
    "This worksheet covers parts of [the Classification II chapter](https://datasciencebook.ca/classification2.html) of the online textbook. You should read this chapter before attempting the worksheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b2bf0e24e6922491ca915a749088f6fc",
     "grade": false,
     "grade_id": "cell-e3518ed3d64fb67c",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "### Run this cell before continuing.\n",
    "library(tidyverse)\n",
    "library(repr)\n",
    "library(tidymodels)\n",
    "options(repr.matrix.max.rows = 6)\n",
    "source('tests.R')\n",
    "source('cleanup.R')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e3bb815c573d1e593ccde12f9813ae9c",
     "grade": false,
     "grade_id": "cell-8699740184695c66",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 0.1** Multiple Choice:\n",
    "<br>{points: 1}\n",
    "\n",
    "Before applying k-nearest neighbour to a classification task, we need to scale the data. What is the purpose of this step?\n",
    "\n",
    "A. To help speed up the knn algorithm.\n",
    "\n",
    "B. To convert all data observations to numeric values.\n",
    "\n",
    "C. To ensure all data observations will be on a comparable scale and contribute equal shares to the calculation of the distance between points.\n",
    "\n",
    "D. None of the above.\n",
    "\n",
    "*Assign your answer to an object called `answer0.1`. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`)*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "868196915f89a2cd8b75833d3b31265c",
     "grade": false,
     "grade_id": "cell-59b72939f6fa91c3",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "75a155e993fb8cc210f042093cdedc2d",
     "grade": true,
     "grade_id": "cell-3f57245c4f46b1d0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_0.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "efeb2215f120a2a1b03bc51db3520cc6",
     "grade": false,
     "grade_id": "cell-0ae53a5320742a59",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## 1. Fruit Data Example - (Part II)\n",
    "**Question 1.0** \n",
    "<br>{points: 1}\n",
    "\n",
    "You will recognize a few of the first questions in this worksheet from last week's tutorial. This will help you repeat some of the fundamentals of classification before tackling the later questions in this worksheet, which integrate concepts you learned from this week's material. First, load the file `fruit_data.csv` (found in the data folder) from the previous tutorial into your notebook.\n",
    "\n",
    "`mutate()` the `fruit_name` column such that it is a factor using `as_factor()`.\n",
    "\n",
    "*Assign your data to an object called `fruit_data`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5dec1e49d953d99d5f1175fa39fdce56",
     "grade": false,
     "grade_id": "cell-79fb584a3431ca43",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7c3b4f5dbc24925a81ee52993294d627",
     "grade": true,
     "grade_id": "cell-03b586646cc9ea74",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "96c22def398ef9777be1d806a18c328d",
     "grade": false,
     "grade_id": "cell-9691b09f8cd7044c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "Let's take a look at the first six observations in the fruit dataset. Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f93525a5f4e7f3394f97443382f48b42",
     "grade": false,
     "grade_id": "cell-19eb282d44a51d90",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Run this cell. \n",
    "slice(fruit_data, 1:6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9bbc0b46966cf89cba182cf5cd89213a",
     "grade": false,
     "grade_id": "cell-8859cd0f52f211a7",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "Run the cell below, and find the nearest neighbour based on mass and width to the first observation just by looking at the scatterplot (the first observation has been circled for you)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c595e24a25160d27fc35efb21453f56e",
     "grade": false,
     "grade_id": "cell-3833e64a2fd0000c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Run this cell. \n",
    "options(repr.plot.width=10, repr.plot.height=6)\n",
    "point1 <- c(192, 8.4)\n",
    "point2 <- c(180, 8)\n",
    "point44 <- c(194, 7.2)\n",
    "\n",
    "fruit_data |>  \n",
    "    ggplot(aes(x=mass, \n",
    "               y= width, \n",
    "               colour = fruit_name)) +\n",
    "        labs(x = \"Mass (grams)\",\n",
    "             y = \"Width (cm)\",\n",
    "            colour = 'Name of the Fruit') +\n",
    "        geom_point() +\n",
    "        annotate(\"path\", \n",
    "                 x=point1[1] + 5*cos(seq(0,2*pi,length.out=100)),\n",
    "                 y=point1[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +\n",
    "        annotate(\"text\", x = 183, y =  8.5, label = \"1\") +\n",
    "        theme(text = element_text(size = 20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "52a1e73d0bea41e562332ebf0b42987e",
     "grade": false,
     "grade_id": "cell-63c4bd6e0c9afee9",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 1.1** Multiple Choice:\n",
    "<br>{points: 1}\n",
    "\n",
    "Based on the graph generated, what is the `fruit_name` of the closest data point to the one circled?\n",
    "\n",
    "A. apple\n",
    "\n",
    "B. lemon\n",
    "\n",
    "C. mandarin\n",
    "\n",
    "D. orange\n",
    "\n",
    "*Assign your answer to an object called `answer1.1`. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "50e62322f1288e75b3c923f23152071f",
     "grade": false,
     "grade_id": "cell-b2af8601532fb0b1",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7959e42e6630f4c45df1af4fe4440079",
     "grade": true,
     "grade_id": "cell-3d469d43b37c3c40",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0c818ef9d59ccba8a4a358235c428605",
     "grade": false,
     "grade_id": "cell-10a0cb2de24a557d",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 1.2**\n",
    "<br>{points: 1}\n",
    "\n",
    "Using `mass` and `width`, calculate the distance between the first observation and the second observation.\n",
    "\n",
    "We provide a scaffolding to get you started.\n",
    "\n",
    "*Assign your answer to an object called `fruit_dist_2`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "476e3a965e4688fd2225725b0d5ac7fd",
     "grade": false,
     "grade_id": "cell-b6995fb3f851dfd1",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# ... <- fruit_data |>\n",
    "#    slice(1, 2) |> # We use slice to get the first two rows of the fruit dataset\n",
    "#    select(mass, ...) |>\n",
    "#    dist()\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9d330b012d12e597f3b30526d91dae35",
     "grade": true,
     "grade_id": "cell-b985ee138ab89218",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d2bfb1d68d0f83e30d6b6950cb5c8175",
     "grade": false,
     "grade_id": "cell-1688abd2ff0a21e7",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 1.3**\n",
    "<br>{points: 1}\n",
    "\n",
    "Calculate the distance between the first and the **44th observation** in the `fruit` dataset using the `mass` and `width` variables.\n",
    "\n",
    "You can see from the data frame output from the cell below that **observation 44** has `mass` = 194 g and `width` = 7.2 cm.\n",
    "\n",
    "*Assign your answer to an object called `fruit_dist_44`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "64778b9334133deefe3237fd61eb9a36",
     "grade": false,
     "grade_id": "cell-c08d592f4a1618b2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# Run this cell. \n",
    "\n",
    "filter(fruit_data, row_number() == 44)\n",
    "\n",
    "point1 <- c(192, 8.4)\n",
    "point2 <- c(180, 8)\n",
    "point44 <- c(194, 7.2)\n",
    "\n",
    "fruit_data |>\n",
    "    ggplot(aes(x = mass, \n",
    "               y = width, \n",
    "               colour = fruit_name)) +\n",
    "        labs(x = \"Mass (grams)\",\n",
    "             y = \"Width (cm)\",\n",
    "            colour = 'Name of the Fruit') +\n",
    "        geom_point() +\n",
    "        annotate(\"path\", \n",
    "                 x=point1[1] + 5*cos(seq(0,2*pi,length.out=100)),\n",
    "                 y=point1[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +\n",
    "        annotate(\"text\", x = 183, y =  8.5, label = \"1\") +\n",
    "        annotate(\"path\",\n",
    "                 x=point2[1] + 5*cos(seq(0,2*pi,length.out=100)),\n",
    "                 y=point2[2] + 0.1*sin(seq(0,2*pi,length.out=100))) +\n",
    "        annotate(\"text\", x = 169, y =  8.1, label = \"2\") +\n",
    "        annotate(\"path\",\n",
    "                 x=point44[1] + 5*cos(seq(0,2*pi,length.out=100)),\n",
    "                 y=point44[2]+0.1*sin(seq(0,2*pi,length.out=100))) +\n",
    "        annotate(\"text\", x = 204, y =  7.1, label = \"44\") +\n",
    "        theme(text = element_text(size = 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cb970fe2038ad019cb9a8e4344e9356a",
     "grade": false,
     "grade_id": "cell-43a8d3f7727d3450",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "20917923330b8f3c68240fa65938a894",
     "grade": true,
     "grade_id": "cell-34cfcd13bb1f5097",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f64f5382c5d075b19ba415c9e136c8e5",
     "grade": false,
     "grade_id": "cell-aee58b612565e7f9",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "What do you notice about your answers from **Question 1.2** & **1.3** that you just calculated? Is it what you would expect given the scatter plot above? Why or why not? Discuss this.\n",
    "\n",
    "*Hint: Look at where the observations are on the scatterplot in the cell above this question. What might happen if we changed grams into kilograms to measure the mass?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b084c5d19a158430a9aba43f726fa4b9",
     "grade": false,
     "grade_id": "cell-5fb7c1e62997eb46",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 1.4**\n",
    "<br>{points: 1}\n",
    "\n",
    "From the distance calculation, we see that observation 1 and 44 have a smaller distance than observation 1 and 2. However, if we look at the scatterplot the distance of the first observation to the second observation appears closer than to the 44th observation.\n",
    "\n",
    "Which of the following statements is correct?\n",
    "\n",
    "A. A difference of 12 g in mass between observation 1 and 2 is large compared to a difference of 1.2 cm in width between observation 1 and 44. Consequently, mass will drive the classification results, and width will have less of an effect. Hence, our distance calculation reflects that.\n",
    "\n",
    "B. If we measured mass in kilograms, then we’d get different classification results.\n",
    "\n",
    "C. We should standardize the data so that all variables will be on a comparable scale.\n",
    "\n",
    "D. All of the above.\n",
    "\n",
    "*Assign your answer to an object called `answer1.4`. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4aaa04db4254ad589a402439f0e9c12f",
     "grade": false,
     "grade_id": "cell-4b81c6055a08f87b",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer1.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2db5705d57b29baa45a5510a039d4fcf",
     "grade": true,
     "grade_id": "cell-89c07cb539498da4",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.4()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9504a2efda7af0ec6cf75b0549b58f6f",
     "grade": false,
     "grade_id": "cell-5b9caaa8370fd73e",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 1.5**\n",
    "<br>{points: 1}\n",
    "\n",
    "Scale and center all the variables of the `fruit` dataset and save them as columns in your data table.\n",
    "\n",
    "Save the dataset object and call it `fruit_data_scaled`. Make sure to name the new columns `scaled_*` where * is the old column name (e.g. `scaled_mass`). Do not drop the unscaled columns (`mass`, `width`, `height`, `color_score`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e0fea44cef07fa7724e014ce4a659249",
     "grade": false,
     "grade_id": "cell-d15b464489d4f1a0",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# .. <- fruit_data |> \n",
    "# mutate(scaled_mass = scale(mass, center = TRUE), \n",
    "#        scaled_width = scale(..., ...),\n",
    "#        scaled_height = scale(..., ...),\n",
    "#        scaled_color_score = scale(..., ...))\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_data_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "477736f8bc1def4942f515ceb0cd0ff9",
     "grade": true,
     "grade_id": "cell-1897bc0107b67ecc",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.5()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d8353b551de8d946a616e4c9b870f7c0",
     "grade": false,
     "grade_id": "cell-b6f8f43cbdc8642a",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "\n",
    "**Question 1.6**\n",
    "<br>{points: 1}\n",
    "\n",
    "Let's repeat **Question 1.2** and **1.3** with the scaled variables:\n",
    "\n",
    "- calculate the distance with the scaled mass and width variables between observations 1 and 2\n",
    "- calculate the distances with the scaled mass and width variables between observations 1 and 44\n",
    "\n",
    "After you do this, think about how these distances compared to the distances you computed in **Question 1.2** and **1.3** for the same points.\n",
    "\n",
    "*Assign your answers to objects called `distance_2` and `distance_44` respectively.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "61f35f4c227b0b25d56ede46dc1b3e73",
     "grade": false,
     "grade_id": "cell-7723cc0237258b6c",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "distance_2\n",
    "distance_44"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "71dd1f49872536e853ad5ee75534122a",
     "grade": true,
     "grade_id": "cell-7316457bcda1bf40",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.6()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "67f52f4e83254e5d82de8b5512fb253c",
     "grade": false,
     "grade_id": "cell-6e0a8546a65b79e8",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Randomness and Setting Seeds\n",
    "\n",
    "This worksheet uses functions from the `tidymodels` library, which not only allows us to perform K-nearest neighbour classification, but also allows us to evaluate how well our classification worked. In order to ensure that the steps in the worksheet are reproducible, we need to set a *seed*, i.e., a numerical \"starting value,\" which determines the sequence of random numbers R will generate.\n",
    "\n",
    "Below in many cells we have included a call to `set.seed`. **Do not remove these lines of code**; they are necessary to make sure the autotesting code functions properly.\n",
    "\n",
    "> *The reason we have `set.seed` in so many places is that Jupyter notebooks are organized into cells that can be run out of order. Since things can be run out of order, the exact sequence of random values that is used in each cell is hard to determine, which makes autotesting really difficult.  We had two options: either enforce that you only ever run the code by hitting \"Restart & Run All\" to ensure that we get the same values of randomness each time, or put `set.seed` in a lot of places (we chose the latter). One drawback of calling `set.seed` everywhere is that the numbers that will be generated won't really be random. For the purposes of teaching and learning, that is fine here. But __in your course project and other data analyses outside of this course, you must call `set.seed` only once at the beginning of the analysis, so that your random numbers are actually reasonably random.__*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b233a104af7a8d925b3354da1935bed3",
     "grade": false,
     "grade_id": "cell-0e8089ce4f710660",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## 2. Splitting the data into a training and test set\n",
    "\n",
    "In this exercise, we will be partitioning `fruit_data` into a training (75%) and testing (25%) set using the `tidymodels` package. After creating the test set, we will put the test set away in a lock box and not touch it again until we have found the best k-nn classifier we can make using the training set. We will use the variable `fruit_name` as our class label. \n",
    "\n",
    "\n",
    "**Question 2.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "To create the training and test set, first use the `initial_split` function to split `fruit_data`. Specify you want to use *75%* of the data. For the `strata` argument, place the variable we want to classify, `fruit_name`. Name the object you create `fruit_split`. \n",
    "\n",
    "Next, pass the `fruit_split` object to the `training` and `testing` functions and name your respective objects `fruit_train` and `fruit_test`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b008fa6b5d42c05562f48a86f64272f2",
     "grade": false,
     "grade_id": "cell-c262636f2b9777e3",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Set the seed. Don't remove this!\n",
    "set.seed(3456) \n",
    "\n",
    "# Randomly take 75% of the data in the training set. \n",
    "# This will be proportional to the different number of fruit names in the dataset.\n",
    "\n",
    "#... <- initial_split(..., prop = ..., strata = ...)  \n",
    "#... <- training(...)   \n",
    "#... <- testing(...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_train\n",
    "fruit_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b26c071c7713ad3fe803ab4bd016e677",
     "grade": true,
     "grade_id": "cell-7432078031e563b8",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a9af5f95503e3c7b209dcccd5348ff44",
     "grade": false,
     "grade_id": "cell-9f6c24da5042200c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 2.1** \n",
    "<br> {points: 1}\n",
    "\n",
    "K-nearest neighbors is sensitive to the scale of the predictors so we should do some preprocessing to standardize them. Remember that standardizing involves centering/shifting (subtracting the mean of each variable) and scaling (dividing by its standard deviation). Also remember that standardization is *part of your training procedure*, so you can't use your test data to compute the shift / scale values for each variable. Therefore, you must pass only the training data to your recipe to compute the preprocessing steps. This ensures that our test data does not influence any aspect of our model training. Once we have created the standardization preprocessor, we can then later on apply it separately to both the training and test data sets.\n",
    "\n",
    "For this exercise, let's see if `mass` and `color_score` can predict `fruit_name`. \n",
    "\n",
    "To scale and center the data, first, pass the vector and the predictors to the `recipe` function. Remember to place your vector *before* your predictors. To scale your predictors, use the `step_scale(all_predictors())` function. To center your predictors, use the `step_center(all_predictors())` function.  \n",
    "\n",
    "*Assign your answer to an object called `fruit_recipe`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "71d7cde637c29d8e9acb0197bf50c359",
     "grade": false,
     "grade_id": "cell-aca4cc578b245211",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- recipe(... ~ ... + ... , data = ...) |>\n",
    "#    ... |>\n",
    "#    ...\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_recipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "38ce11b9e4c25a61724ed945c13badcc",
     "grade": true,
     "grade_id": "cell-2453aaa31ba794b6",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ff6abff7b1910f8b822605d9bbb6fdbf",
     "grade": false,
     "grade_id": "cell-6693330a0f077083",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 2.2**\n",
    "<br> {points: 1}\n",
    "\n",
    "So far, we have split the training and testing datasets as well as preprocessed the data. Now, let's create our K-nearest neighbour classifier with only the training set using the `tidymodels` package. First, create the classifier by specifying that we want $K = 3$ neighbors and that we want to use the *straight-line* distance. \n",
    "\n",
    "*Assign your answer to an object called `knn_spec`*.  \n",
    "\n",
    "Next, train the classifier with the training data set using the `workflow` function. This function allows you to bundle together your pre-processing, modeling, and post-processing requests. Scaffolding is provided below for you.\n",
    "\n",
    "*Assign your answer to an object called `fruit_fit`*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1622936b33fa7c94944d3c0189353368",
     "grade": false,
     "grade_id": "cell-00b2a6b0ba7df6c4",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- nearest_neighbor(weight_func = ..., neighbors = ...) |>\n",
    "#       set_engine(...) |>\n",
    "#       set_mode(...)\n",
    "\n",
    "#... <- workflow() |>\n",
    "#       add_recipe(...) |>\n",
    "#       add_model(...) |>\n",
    "#       fit(data = ...)\n",
    "\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e7f635299ba4528a5cf607a17c0560af",
     "grade": true,
     "grade_id": "cell-b8ce3699299555d5",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "de81da7b3b67f3e00a430dbaeb396409",
     "grade": false,
     "grade_id": "cell-512f0c24a1511248",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 2.3**\n",
    "<br> {points: 1}\n",
    "\n",
    "Now that we have created our K-nearest neighbor classifier object, let's predict the class labels for our test set.\n",
    "\n",
    "First, pass your fitted model and the **test dataset** to the `predict` function. Then, use the `bind_cols` function to add the column of predictions to the original test data. \n",
    "\n",
    "*Assign your answer to an object called `fruit_test_predictions`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9c8a59077ea7d15d61096ca9ca2b74dd",
     "grade": false,
     "grade_id": "cell-29a65e0d69fd64d3",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- predict(... , ...) |>\n",
    "#       ...(...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f76e36541eb840d9aa116186c3d19d12",
     "grade": true,
     "grade_id": "cell-144f33352d2b6add",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "962e83775c37ad3e549fa36b5e70462d",
     "grade": false,
     "grade_id": "cell-baa81bf7577a344f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 2.4**\n",
    "<br> {points: 1}\n",
    "\n",
    "Great! We have now computed some predictions for our test datasets! From glancing at the dataframe above, it looks like most of them are correct, but wouldn't it be interesting if we could find out our classifier's accuracy? \n",
    "\n",
    "Thankfully, the `metrics` function from the `tidymodels` package can help us. To get the statistics about the quality of our model, you need to specify the `truth` and `estimate` arguments. In the `truth` argument, you should put the column name for the true values of the response variable. In the `estimate` argument, you should put the column name for response variable predictions. \n",
    "\n",
    "*Assign your answer to an object called `fruit_prediction_accuracy`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ee126d204fa9f564ae7d5fa0152fe763",
     "grade": false,
     "grade_id": "cell-4ac19dfcaec7e27c",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- fruit_test_predictions |>\n",
    "#         ...(truth = ..., estimate = ...)             \n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_prediction_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6518c3556f30d94035a357daab620cb6",
     "grade": true,
     "grade_id": "cell-a268da3370127d6c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.4()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "85b92a9f2dce2d1223c1ce1b498af911",
     "grade": false,
     "grade_id": "cell-67a957c51db2a297",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 2.5**\n",
    "<br> {points: 1}\n",
    "\n",
    "Now, let's look at the *confusion matrix* for the classifier. This will show us a table comparing the predicted labels with the true labels. \n",
    "\n",
    "A confusion matrix is essentially a classification matrix. The columns of the confusion matrix represent the actual class and the rows represent the predicted class (or vice versa). Shown below is an example of a confusion matrix.\n",
    "\n",
    "|                  |          |  Actual Values |                |\n",
    "|:----------------:|----------|:--------------:|:--------------:|\n",
    "|                  |          |    Positive    |    Negative    |\n",
    "|**Predicted Value**  | Positive |  True Positive | False Positive|\n",
    "|                  | Negative | False Negative | True Negative  |\n",
    "\n",
    "\n",
    "- A **true positive** is an outcome where the model correctly predicts the positive class.\n",
    "- A **true negative** is an outcome where the model correctly predicts the negative class.\n",
    "- A **false positive** is an outcome where the model incorrectly predicts the positive class.\n",
    "- A **false negative** is an outcome where the model incorrectly predicts the negative class.\n",
    "\n",
    "<br>\n",
    "\n",
    "We can create a confusion matrix by using the `conf_mat` function. Similar to the `metrics` function, you will have to specify the `truth` and `estimate` arguments. \n",
    "\n",
    "*Assign your answer to an object called `fruit_mat`*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f24e7fdd4cf6febe4d9718342d5b04a4",
     "grade": false,
     "grade_id": "cell-f0716222fa3e3c93",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- fruit_test_predictions |> \n",
    "#       ...(truth = ..., estimate = ...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "50ba35a8c51cc1a58b6bdbbb2dcfbda0",
     "grade": true,
     "grade_id": "cell-d3d5c46250e8450a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.5()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ffc78c73873a943eef6ee4b2d880157d",
     "grade": false,
     "grade_id": "cell-76907faaa508f23b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 2.6** Multiple Choice:\n",
    "<br> {points: 1}\n",
    "\n",
    "Reading `fruit_mat`, how many observations were labelled correctly?\n",
    "\n",
    "A. 14\n",
    "\n",
    "B. 12\n",
    "\n",
    "C. 10\n",
    "\n",
    "D. 13\n",
    "\n",
    "*Assign your answer to an object called `answer2.6`. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d7dc42422b7bc986c7a575478f241a04",
     "grade": false,
     "grade_id": "cell-c4d8f09cbf40df59",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer2.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2a8cb4af39a241cd0760105114251180",
     "grade": true,
     "grade_id": "cell-5715f106e3bbf143",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.6()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7df408afe47dc0f564a96c403fbd4b51",
     "grade": false,
     "grade_id": "cell-0063191266f982c7",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "### 3. Cross-validation\n",
    "\n",
    "**Question 3.1**\n",
    "<br> {points: 1}\n",
    "\n",
    "The vast majority of predictive models in statistics and machine learning have parameters that you have to pick. For the past few exercises, we have had to pick the number of neighbours for the class vote, which we have done arbitraily. But, is it possible to make this selection, *i.e., tune the model, in a principled way?* Ideally, we want to pick the number of neighborurs to maximize the performance of our classifier on data *it hasn’t seen yet*.\n",
    "\n",
    "An important aspect of the tuning process is that we can, if we want to, split our training data again, train and evaluate a classifier for each split, and then choose the parameter based on all of the different results. If we just split our training data once, our best parameter choice will depend strongly on the randomness from how this single split was made. Using multiple different splits, we’ll get a more robust estimate of accuracy, which will lead to a more suitable choice of the number of neighbours $K$ to perform well on unseen data.\n",
    "\n",
    "The idea of training and evaluating models on multiple training data splits times is called \"cross-validation\". In cross-validation, we split our overall training data into $C$ evenly-sized chunks, and then iteratively use 1 chunk as the **validation set** and combine the remaining $C−1$ chunks as the **training set.** The validation set is used in a similar was as the test set, **except** that the test set is only used once at the end to report model performance whether we use model performance on the validation set to select the model during cross-validation. \n",
    "\n",
    "---\n",
    "\n",
    "We can perform a cross-validation in R using the `vfold_cv` function. To use this function, you have to identify the training set as well as specify the `v` (the number of folds) and the `strata` argument (the label variable). \n",
    "\n",
    "For this exercise, perform **$5$-fold cross-validation**.\n",
    "\n",
    "*Assign your answer to an object called `fruit_vfold`*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "afdcc7d9663ecfdcb036296b6ebbe466",
     "grade": false,
     "grade_id": "cell-40206dc81c114f74",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- vfold_cv(..., v = ..., strata = ...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b591b806779aacafd2bb2a3e4511ae01",
     "grade": true,
     "grade_id": "cell-4f7b62f0f7b0a200",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_3.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "4d1e589450a8e1099c29da82263525f7",
     "grade": false,
     "grade_id": "cell-068da1c1156e503a",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 3.2**\n",
    "<br> {points: 1}\n",
    "\n",
    "Now perform the workflow analysis again. You can reuse the `fruit_recipe` and `knn_spec` objects you made earlier. When you are fitting the knn model, use the `fit_resamples` function instead of the `fit` function for training. This function will allow us to run a cross-validation on each train/validation split we created in the previous question. \n",
    "\n",
    "*Assign your answer to an object called `fruit_resample_fit`*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b67c2a89cd7ef78fb5be1bf95859d72e",
     "grade": false,
     "grade_id": "cell-0474eb947592495c",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- workflow() |>\n",
    "#       add_recipe(...) |>\n",
    "#       add_model(...) |>\n",
    "#       fit_resamples(resamples = ...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2f8e388622fdde399326dcbb41f693bd",
     "grade": true,
     "grade_id": "cell-099fb002dba6ced0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_3.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "40bdb9b98b1c06b579eb2d8f5e162461",
     "grade": false,
     "grade_id": "cell-df3dc35f0cbb35e2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 3.3** \n",
    "<br> {points: 1}\n",
    "\n",
    "Now that we have ran a cross-validation on each train/validation split, one has to ask, how accurate was the classifier's validation across the folds? We can aggregate the *mean* and *standard error* by using the `collect_metrics` function. The standard error is essentially a measure of how uncertain we are in the mean value. \n",
    "\n",
    "Use the `collect_metrics` function on the `fruit_resample_fit` object and assign your answer to an object called `fruit_metrics`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "aa2b50e63553e0fab6b486d57fad0794",
     "grade": false,
     "grade_id": "cell-1119165a08beb72d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "fruit_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ff6f588520207e25a68b527978186260",
     "grade": true,
     "grade_id": "cell-7337cd2a0af4b896",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_3.3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5939bdb0bd8c070ee2404571f6f32a2b",
     "grade": false,
     "grade_id": "cell-b65f60bbbc0a4884",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## 4. Parameter value selection\n",
    "\n",
    "Using a 5-fold cross-validation, we have established a prediction accuracy for our classifier. \n",
    "\n",
    "If we had to improve our classifier, we have to change the parameter: number of neighbours, $K$. Since cross-validation helps us evaluate the accuracy of our classifier, we can use cross-validation to calculate an accuracy for each value of $K$ in a reasonable range, and then pick the value of $K$ that gives us the best accuracy. \n",
    "\n",
    "The great thing about the `tidymodels` package is that it provides a very simple syntax for tuning models. Using `tune()`, each parameter in the model can be adjusted rather than given a specific value. \n",
    "\n",
    "**Question 4.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "Create a new K-nearest neighbor model specification but instead of specifying a particular value for the `neighbors` argument, insert `tune()`. \n",
    "\n",
    "*Assign your answer to an object called `knn_tune`.* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e4d8871755f64d87e26be17856b5fea5",
     "grade": false,
     "grade_id": "cell-2d0448675438bdec",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "knn_tune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ccc6b19b3179ffd5a1ba1b251fcfb151",
     "grade": true,
     "grade_id": "cell-381d21c4c1474e80",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_4.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a44cafde61efbc0f0e75bb30bab2612d",
     "grade": false,
     "grade_id": "cell-706cde9c0d8d2699",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 4.1**\n",
    "<br>{points: 1}\n",
    "\n",
    "Now, create a `workflow()` analysis that combines `fruit_recipe` and our new `knn_tune` model specification. \n",
    "\n",
    "Instead of using `fit` or `fit_resamples`, we will use the `tune_grid` function to fit the model for each value in a range of parameter values. For the `resamples` argument, input the cross-validation `fruit_vfold` model we created earlier. The `grid` argument specifies that the tuning should try $X$ amount of values of the number of neighbors $K$ when tuning. For this exercise, use *10 $K$ values when tuning*. \n",
    "\n",
    "Finally, aggregate the mean and standard error by using the `collect_metrics` function.\n",
    "\n",
    "*Assign your answer to an object called `knn_results`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2bcc2de28443d2365c46fc4535d5de7d",
     "grade": false,
     "grade_id": "cell-c0d2f1edbe342214",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "set.seed(1234) # set the seed, don't remove this\n",
    "\n",
    "#... <- workflow() |>\n",
    "#       add_recipe(...) |>\n",
    "#       add_model(...) |>\n",
    "#       tune_grid(resamples = ..., grid = ...) |>\n",
    "#       ...\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "knn_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e2d9381f3f1c945a4f208684ea04e6b5",
     "grade": true,
     "grade_id": "cell-0cbc6a47a2b14884",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_4.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2a60ce3465536d0a1c1c38957b89569c",
     "grade": false,
     "grade_id": "cell-e8a71fe7a9789225",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 4.2**\n",
    "<br>{points: 1} \n",
    "\n",
    "\n",
    "Now, let's find the best value of the number of neighbors. \n",
    "\n",
    "First, from `knn_results`, filter for `accuracy` from the `.metric` column. \n",
    "\n",
    "*Assign your answer to an object called `accuracies`.*\n",
    "\n",
    "Next, create a line plot using the `accuracies` dataset with `neighbors` on the x-axis and the `mean` on the y-axis. \n",
    "\n",
    "*Assign your answer to an object called `accuracy_versus_k`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c1de51d8f96cc10a0e548c35636e24a9",
     "grade": false,
     "grade_id": "cell-b8ae5cb978d3f8f2",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "#... <- knn_results |> \n",
    "#       filter(...)\n",
    "\n",
    "#... <- ggplot(..., aes(x = ..., y = ...))+\n",
    "#       geom_point() +\n",
    "#       geom_line() +\n",
    "#       labs(x = \"Neighbors\", y = \"Accuracy Estimate\") +\n",
    "#       scale_x_continuous(breaks = seq(0, 14, by = 1)) +  # adjusting the x-axis\n",
    "#       scale_y_continuous(limits = c(0.4, 1.0)) # adjusting the y-axis\n",
    "\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "accuracy_versus_k"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d36c5097a082447e87d5b791da2a3696",
     "grade": true,
     "grade_id": "cell-6d62d960eaff34d6",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_4.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "55568644264b3cb3301d7ae175ae5cc8",
     "grade": false,
     "grade_id": "cell-ff9e1c1f4f70c1e9",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "From the plot above, we can see that $K = 2$, $3$, or $4$ provides the highest accuracy. Larger $K$ values result in a reduced accuracy estimate. Remember: the values you see on this plot are estimates of the true accuracy of our classifier. Although the $K = 2$, $3$ or $4$ value is higher than the others on this plot, that doesn’t mean the classifier is necessarily more accurate with this parameter value! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b96cf43f98ea9bf9e2c9958a497023ff",
     "grade": false,
     "grade_id": "cell-0b119208147a50fc",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "Great, now you have completed a full workflow analysis with cross-validation using the `tidymodels` package! For your information, we can choose any number of folds and typically, the more we use the better our accuracy estimate will be (lower standard error). However, more folds would mean a greater computation time. In practice, $C$ is chosen to be either 5 or 10. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b1e27fcdb53cd0811a4aebaba6c3c824",
     "grade": false,
     "grade_id": "cell-4609bfd4722e6480",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "source('cleanup.R')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}