worksheet_regression1/worksheet_regression1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b8f32d680fd99e0e33281247976a7eac",
     "grade": false,
     "grade_id": "cell-6093d584be2ad9ed",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "# Worksheet 8 - Regression\n",
    "\n",
    "### Lecture and Tutorial Learning Goals:\n",
    "\n",
    "After completing this week's lecture and tutorial work, you will be able to:\n",
    "- Recognize situations where a simple regression analysis would be appropriate for making predictions.\n",
    "- Explain the k-nearest neighbour ($k$-nn) regression algorithm and describe how it differs from $k$-nn classification.\n",
    "- Interpret the output of a $k$-nn regression.\n",
    "- In a dataset with two variables, perform k-nearest neighbour regression in R using `tidymodels` to predict the values for a test dataset.\n",
    "- Using R, execute cross-validation in R to choose the number of neighbours.\n",
    "- Using R, evaluate $k$-nn regression prediction accuracy using a test data set and an appropriate metric (e.g., root mean square prediction error, RMSPE).\n",
    "- In the context of $k$-nn regression, compare and contrast goodness of fit and prediction properties (namely RMSE vs RMSPE).\n",
    "- Describe advantages and disadvantages of the $k$-nearest neighbour regression approach.\n",
    "\n",
    "This worksheet covers parts of [the Regression I chapter](https://datasciencebook.ca/regression1.html) of the online textbook. You should read this chapter before attempting the worksheet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2cb5e5a9a2f495df08f646e0768e0fc2",
     "grade": false,
     "grade_id": "cell-53085d5554f9dcc8",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "### Run this cell before continuing.\n",
    "library(tidyverse)\n",
    "library(repr)\n",
    "library(tidymodels)\n",
    "options(repr.matrix.max.rows = 6)\n",
    "source(\"tests.R\")\n",
    "source('cleanup.R')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "004a90536551b93c7af8a8b5d3e92957",
     "grade": false,
     "grade_id": "cell-8e40e06455324a19",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 0.0** Multiple Choice: \n",
    "<br> {points: 1}\n",
    "\n",
    "To predict a value of $Y$ for a new observation using $k$-nn **regression**, we identify the $k$-nearest neighbours and then:\n",
    "\n",
    "A. Assign it the median of the $k$-nearest neighbours as the predicted value\n",
    "\n",
    "B. Assign it the mean of the $k$-nearest neighbours as the predicted value\n",
    "\n",
    "C. Assign it the mode of the $k$-nearest neighbours as the predicted value\n",
    "\n",
    "D. Assign it the majority vote of the $k$-nearest neighbours as the predicted value\n",
    "\n",
    "*Save the letter of the answer you think is correct to a variable named `answer0.0`. Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b2a94d99e15ed916391daed3643fe923",
     "grade": false,
     "grade_id": "cell-108236d99368cc44",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e245528180447ef32eb2e47de0ac531e",
     "grade": true,
     "grade_id": "cell-7d5938e33a38edeb",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_0.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fe13b300b073467ed185bbcb049aa62b",
     "grade": false,
     "grade_id": "cell-b0f4fa2237ef0429",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 0.1** Multiple Choice: \n",
    "<br> {points: 1}\n",
    "\n",
    "Of those shown below, which is the correct formula for RMSPE?\n",
    "\n",
    "\n",
    "A. $RMSPE = \\sqrt{\\frac{\\frac{1}{n}\\sum\\limits_{i=1}^{n}(y_i - \\hat{y_i})^2}{1 - n}}$\n",
    "\n",
    "B. $RMSPE = \\sqrt{\\frac{1}{n - 1}\\sum\\limits_{i=1}^{n}(y_i - \\hat{y_i})^2}$\n",
    "\n",
    "C. $RMSPE = \\sqrt{\\frac{1}{n}\\sum\\limits_{i=1}^{n}(y_i - \\hat{y_i})^2}$\n",
    "\n",
    "D. $RMSPE = \\sqrt{\\frac{1}{n}\\sum\\limits_{i=1}^{n}(y_i - \\hat{y_i})}$ \n",
    "\n",
    "*Save the letter of the answer you think is correct to a variable named `answer0.1`. Make sure you put quotations around the letter and pay attention to case.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5fd4ab0d451691ecaad16cf85eeeeefc",
     "grade": false,
     "grade_id": "cell-93c565d6974de38f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f7b0eb36bc88d416ff71c8abeaa9227e",
     "grade": true,
     "grade_id": "cell-b1e66491132c1e30",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_0.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "bcdce0f1b28093249b5eec7e4e5dd637",
     "grade": false,
     "grade_id": "cell-ded5f8fa4e621b81",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 0.2**\n",
    "<br> {points: 1}\n",
    "\n",
    "The plot below is a very simple k-nn regression example, where the black dots are the data observations and the blue line is the predictions from a $k$-nn regression model created from this data where $k=2$.\n",
    "\n",
    "Using the formula for RMSE (given in the reading), and the graph below, by hand (pen and paper or use R as a calculator) calculate RMSE for this model. Use **one decimal place of precision** when inputting the heights of the black dots and blue line. Save your answer to a variable named `answer0.2` \n",
    "\n",
    "*Note: The predicted value when x = 1 is 1.3 (it's a bit hard to tell from the figure!)*\n",
    "\n",
    "<img align=\"left\" src=\"img/k-nn_RMSE.jpeg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "67e086cc04ad4264b642781e233c7cd7",
     "grade": false,
     "grade_id": "cell-8dc7cc6f35de80cb",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cd327e469d66107c2a4ca6fc92b97eba",
     "grade": true,
     "grade_id": "cell-d90b3e7b890ce86b",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_0.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5ae59861171794c4f66b5fc614187d8e",
     "grade": false,
     "grade_id": "cell-73d7913554df0781",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### RMSPE Definition\n",
    "\n",
    "**Question 0.3** Multiple Choice: \n",
    "<br> {points: 1}\n",
    "\n",
    "What does RMSPE stand for?\n",
    "\n",
    "\n",
    "A. root mean squared prediction error\n",
    "\n",
    "B. root mean squared percentage error \n",
    "\n",
    "C. root mean squared performance error \n",
    "\n",
    "D. root mean squared preference error \n",
    "\n",
    "*Save the letter of the answer you think is correct to a variable named `answer0.3`. Make sure you put quotations around the letter and pay attention to case.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a1690d2ede99022f3136074dccd5e638",
     "grade": false,
     "grade_id": "cell-81be027da315cb3f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer0.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3ff0cac1a6dea5c2b79a238504979c0e",
     "grade": true,
     "grade_id": "cell-db93265fd197f97f",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_0.3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c94e4bc2ed7206d48d7655fae0781008",
     "grade": false,
     "grade_id": "cell-e612110169987c7b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Marathon Training\n",
    "\n",
    "<img src='https://media.giphy.com/media/nUN6InE2CodRm/giphy.gif' width='400'>\n",
    "\n",
    "Source: https://media.giphy.com/media/nUN6InE2CodRm/giphy.gif\n",
    "\n",
    "What predicts which athletes will perform better than others? Specifically, we are interested in marathon runners, and looking at how the maximum distance ran per week (in miles) during race training predicts the time it takes a runner to finish the race? For this, we will be looking at the `marathon.csv` file in the `data/` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9ea30c1434a67e3876226cd543d43d4c",
     "grade": false,
     "grade_id": "cell-16d7d432c49c9e53",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 1.0** \n",
    "<br> {points: 1}\n",
    "\n",
    "Load the data and assign it to an object called `marathon`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e808fb877b552ca26971fdb18432c347",
     "grade": false,
     "grade_id": "cell-5376dc2c39983258",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "458c6058a4988dffc4082aca21145987",
     "grade": true,
     "grade_id": "cell-60554f6f6a56a16c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_1.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e60d5964d61d5758585592dfb52db3b2",
     "grade": false,
     "grade_id": "cell-26802c14f46a4d8e",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 2.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "We want to predict race time (in hours) (`time_hrs`) given a particular value of maximum distance ran per week (in miles) during race training (`max`). Let's take a subset of size 50 individuals of our marathon data and assign it to an object called `marathon_50`. With this subset, plot a scatterplot to assess the relationship between these two variables. Put `time_hrs` on the y-axis and `max` on the x-axis. **Assign this plot to an object called `answer2`.** Discuss, with a classmate, the relationship between race time and maximum distance ran per week during training based on the scatterplot you create below. \n",
    "\n",
    "*Hint: To take a subset of your data you can use the `sample_n()` function*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "519a49b25c72539a147e52979383207c",
     "grade": false,
     "grade_id": "cell-6fcfc6ca32883f4d",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "options(repr.plot.width = 8, repr.plot.height = 7)\n",
    "\n",
    "set.seed(2000) ### DO NOT CHANGE\n",
    "\n",
    "#... <- ... |>\n",
    "#    sample_n(...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b77a51825a4682065b05a3e849538c39",
     "grade": true,
     "grade_id": "cell-78ac50fb41cd9a86",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_2.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "34f89317595dbfcb55227675d68dc3a2",
     "grade": false,
     "grade_id": "cell-cbe9f9a5c4b7c264",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 3.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "Suppose we want to predict the race time for someone who ran a maximum distance of 100 miles per week during training. In the plot below we can see that no one has run a maximum distance of 100 miles per week. But, if we are interested in prediction, how can we predict with this data? We can use $k$-nn regression! To do this we get the $Y$ values (target/response variable) of the nearest $k$ values and then take their average and use that as the prediction. \n",
    "\n",
    "For this question predict the race time based on the 4 closest neighbors to the 100 miles per week during training.\n",
    "\n",
    "*Fill in the scaffolding below and assign your answer to an object named `answer3`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "75efaaf8fe04ae9b8a48d555d95b030b",
     "grade": false,
     "grade_id": "cell-51d4b16bee915d31",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# run this cell to see a visualization of the 4 nearest neighbours\n",
    "options(repr.plot.height = 6, repr.plot.width = 7)\n",
    "marathon_50 |>\n",
    "    ggplot(aes(x = max, y = time_hrs)) + \n",
    "        geom_point(color = 'dodgerblue', alpha = 0.4) +\n",
    "        geom_vline(xintercept = 100, linetype = \"dotted\") +\n",
    "        xlab(\"Maximum Distance Ran per \\n Week During Training (mi)\") +\n",
    "        ylab(\"Race Time (hours)\") + \n",
    "        geom_segment(aes(x = 100, y = 2.56, xend = 107, yend = 2.56), col = \"orange\") +\n",
    "        geom_segment(aes(x = 100, y = 2.65, xend = 90, yend = 2.65), col = \"orange\") +\n",
    "        geom_segment(aes(x = 100, y = 2.99, xend = 86, yend = 2.99), col = \"orange\") +\n",
    "        geom_segment(aes(x = 100, y = 3.05, xend = 82, yend = 3.05), col = \"orange\") +\n",
    "        theme(text = element_text(size = 20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4babff4645a19226f94b42c479b7645c",
     "grade": false,
     "grade_id": "cell-a8d1b15c312d9c3d",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "#... <- ... |> \n",
    "#  mutate(diff = abs(100 - ...)) |> \n",
    "#  ...(diff, ...) |> \n",
    "#  summarise(predicted = ...(...)) |>\n",
    "#  pull()\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5009d7846f565e8a8bc680274fa8800b",
     "grade": true,
     "grade_id": "cell-af7afd4794001c7b",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_3.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b1f9aa206f133969fcff7bf2009bbd0e",
     "grade": false,
     "grade_id": "cell-0c2107e7f52efb3d",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 4.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "For this question, let's instead predict the race time based on the 2 closest neighbors to the 100 miles per week during training.\n",
    "\n",
    "*Assign your answer to an object named `answer4`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3371f7bb2081a1f1eb48c702eebd05b6",
     "grade": false,
     "grade_id": "cell-44994bbd299ff0d0",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "583a4300b1df6b60f37f8ac9470137e8",
     "grade": true,
     "grade_id": "cell-690e322810064165",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_4.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a7b5791bb4f841adb405ff5f20f2fc83",
     "grade": false,
     "grade_id": "cell-3311ba44befa3767",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 5.0** Multiple Choice:\n",
    "<br> {points: 1}\n",
    "\n",
    "So far you have calculated the $k$ nearest neighbors predictions manually based on $k$'s we have told you to use. However, last week we learned how to use a better method to choose the best $k$ for classification. \n",
    "\n",
    "Based on what you learned last week and what you have learned about $k$-nn regression so far this week, which method would you use to choose the $k$ (in the situation where we don't tell you which $k$ to use)?\n",
    "\n",
    "- A) Choose the $k$ that excludes most outliers\n",
    "- B) Choose the $k$ with the lowest training error\n",
    "- C) Choose the $k$ with the lowest cross-validation error\n",
    "- D) Choose the $k$ that includes the most data points\n",
    "- E) Choose the $k$ with the lowest testing error\n",
    "\n",
    "*Assign your answer to an object called `answer5`.  Make sure your answer is an uppercase letter and is surrounded by quotation marks (e.g. `\"F\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "fe8b9cbbbd4aa881d0e9ab40c0a0b96e",
     "grade": false,
     "grade_id": "cell-01b498d6b1415bf5",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "answer5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a6e6dbf472457ba6aaecfaaff0437f27",
     "grade": true,
     "grade_id": "cell-7cbe965dbdb7228d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_5.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1ac1a0e15ea3ffa70b2260b0aa682b40",
     "grade": false,
     "grade_id": "cell-b3b34499fe540e49",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 6.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "We have just seen how to perform k-nn regression manually, now we will apply it to the whole dataset using the `tidymodels` package. To do so, we will first need to create the training and testing datasets. Split the data using *75%* of the `marathon` data as your training set and set `time_hrs` as the `strata` argument. Store this data into an object called `marathon_split`. \n",
    "\n",
    "Then, use the appropriate `training` and `testing` functions to create your training set which you will call `marathon_training` and your testing set which you will call `marathon_testing`. Remember we won't touch the test dataset until the end. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f1bfffda9ee38c9d79c21ae78613aac4",
     "grade": false,
     "grade_id": "cell-8257a9338314019e",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2000) ### DO NOT CHANGE\n",
    "\n",
    "#... <- initial_split(..., prop = ..., strata = ...)\n",
    "#... <- training(...)\n",
    "#... <- testing(...)\n",
    "\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "25046704f1a619c093f639895b03bc5b",
     "grade": true,
     "grade_id": "cell-381c8aeacf15fd35",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_6.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "74ff1bb05f9e4c9cef9bc64375ead59a",
     "grade": false,
     "grade_id": "cell-0c01c0b6cf4d8e91",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 7.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "Next, we’ll use cross-validation on our **training data** to choose $k$. In $k$-nn classification, we used accuracy to see how well our predictions matched the true labels. In the context of $k$-nn *regression*, we will use RMPSE instead. Interpreting the RMPSE value can be tricky but generally speaking, if the prediction values are very close to the true values, the RMSPE will be small. Conversely, if the prediction values are *not* very close to the true values, the RMSPE will be quite large. \n",
    "\n",
    "Let's perform a cross-validation and choose the optimal $k$. First, create a model specification for $k$-nn. We are still using the $k$-nearest neighbours algorithm, and so we will still use the same package for the model engine as we did in classification (`\"kknn\"`). As usual, specify that we want to use the *straight-line distance*. However, since this will be a regression problem, we will use `set_mode(\"regression\")` in the model specification. Store your model specification in an object called `marathon_spec`. \n",
    "\n",
    "Moreover, create a recipe to preprocess our data. Store your recipe in an object called `marathon_recipe`. The recipe should specify that the response variable is race time (hrs) and the predictor is maximum distance ran per week during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "31868ce196b9c3e528132deaef193f83",
     "grade": false,
     "grade_id": "cell-766f1e094dd32efc",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(1234) #DO NOT REMOVE\n",
    "\n",
    "#... <- nearest_neighbor(weight_func = ..., neighbors = ...) |> \n",
    "#       set_engine(...) |>\n",
    "#       set_mode(...) \n",
    "\n",
    "#... <- recipe(... ~ ..., data = ...) |>\n",
    "#       step_scale(...) |>\n",
    "#       step_center(...)\n",
    "#  \n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_recipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "81e32ccc9a1601d188f1378e8aecfc6d",
     "grade": true,
     "grade_id": "cell-289bc7a299c4f482",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_7.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b591bde125171c97afe36a1a6c5a527b",
     "grade": false,
     "grade_id": "cell-9bbc49f6977aa3fc",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 7.1**\n",
    "<br>{points: 1}\n",
    "\n",
    "Now, create the splits for cross-validation with *5 folds* using the `vfold_cv` function. Store your answer in an object called `marathon_vfold`. Make sure to set the `strata` argument.\n",
    "\n",
    "Then, use the `workflow` function to combine your model specification and recipe. Store your answer in an object called `marathon_workflow`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c551fd220ee0f022dd4d1aa49e0a74b9",
     "grade": false,
     "grade_id": "cell-7cf222d5ae9a2f89",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "set.seed(1234) # DO NOT REMOVE\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5d53d69530adfeb056d80929692536d9",
     "grade": true,
     "grade_id": "cell-88d9f578265d6294",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_7.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "cfca69699d0397fe5d67d13b0e549f0c",
     "grade": false,
     "grade_id": "cell-7cd5193eecc27c30",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 8.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "If you haven't noticed by now, the major difference compared to other workflows from Chapters 6 and 7 is that we are running a *regression* rather than a *classification*. Specifying *regression* in the `set_mode` function essentially tells `tidymodels` that we need to use different metrics (RMPSE rather than accuracy) for tuning and evaluation. \n",
    "\n",
    "Now, let's use the RMSPE to find the best setting for $k$ from our workflow. Let's test the values $k = 1, 11, 21, 31, ..., 81$.\n",
    "\n",
    "First, create a tibble with a column called `neighbors` that contains the sequence of values. Remember that you should use the `seq` function to create the range of $k$s that goes *from* 1 *to* 81 *by jumps of* 10.  Assign that tibble to an object called `gridvals`. \n",
    "\n",
    "Next, tune your workflow such that it tests all the values in `gridvals` and *resamples* using your cross-validation data set. Finally, collect the statistics from your model. Assign your answer to an object called `marathon_results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "79b57741a3f0d43055e78fc0f333749c",
     "grade": false,
     "grade_id": "cell-8c7e7ded673d28ec",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2019) # DO NOT CHANGE\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7c81641a87fa6819b68973bf5315d3c1",
     "grade": true,
     "grade_id": "cell-29d608ebfdd11366",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_8.0()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ffc7314876aa53957c34b549d51a8367",
     "grade": false,
     "grade_id": "cell-35eaea9c384c5f6c",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 8.1**\n",
    "<br> {points: 1}\n",
    "\n",
    "Great! Now find the *minimum* RMSPE along with it's associated metrics such as the mean and standard error, to help us find the number of neighbors that will serve as our best $k$ value. Your answer should simply be a tibble with one row. Assign your answer to an object called `marathon_min`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "99a6de5bf9d8d8c29eb0df4704b4e848",
     "grade": false,
     "grade_id": "cell-f9738d45da9e8a77",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2020) # DO NOT REMOVE\n",
    "\n",
    "#... <- marathon_results |>\n",
    "#    filter(.metric == ...) |>\n",
    "#    slice_min(..., ...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_min"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f6e36629a5557baacb615205df90e6f7",
     "grade": true,
     "grade_id": "cell-a1e34e1fb62d79f5",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_8.1()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "059df927a68d576994b89a53009dd594",
     "grade": false,
     "grade_id": "cell-28a414eb59f5ae20",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 8.2**\n",
    "<br> {points: 1}\n",
    "\n",
    "To assess how well our model might do at predicting on unseen data, we will assess its RMSPE on the test data. To do this, we will first re-train our $k$-nn regression model on the entire training data set, using the $K$ value we obtained from **Question 8.1**. \n",
    "\n",
    "To start, pull the best `neighbors` value from `marathon_min` and store it an object called `k_min`. \n",
    "\n",
    "Following that, we will repeat the workflow analysis again but with a brand new model specification with `k_min`. Remember, we are doing a regression analysis, so please select the appropriate mode. Store your new model specification in an object called `marathon_best_spec` and your new workflow analysis in an object called `marathon_best_fit`. You can reuse this `marathon_recipe` for this workflow, as we do not need to change it for this task.\n",
    "\n",
    "Then, we will use the `predict` function to make predictions on the test data, and use the `metrics` function again to compute a summary of the regression's quality. Store your answer in an object called `marathon_summary`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8d9c278960e85de2b5b718d977d715cd",
     "grade": false,
     "grade_id": "cell-a4de6046a0bd5f96",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(1234) # DO NOT REMOVE\n",
    "\n",
    "#... <- marathon_min |>\n",
    "#          pull(...)\n",
    "\n",
    "#... <- nearest_neighbor(weight_func = ..., neighbors = ...) |>\n",
    "#          set_engine(...) |>\n",
    "#          set_mode(...)\n",
    "\n",
    "#... <- workflow() |>\n",
    "#          add_recipe(...) |>\n",
    "#          add_model(...) |>\n",
    "#          fit(data = ...)\n",
    "\n",
    "#... <- marathon_best_fit |>\n",
    "#           predict(...) |>\n",
    "#           bind_cols(...) |>\n",
    "#           metrics(truth = ..., estimate = ...)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bf6b13b252a8b4c0bb2fb63a8e8623e3",
     "grade": true,
     "grade_id": "cell-94fab75dca459b65",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_8.2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1feec2b034696ca26148ef224854254e",
     "grade": false,
     "grade_id": "cell-dffef3173aff9b72",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "What does this RMSPE score mean? RMSPE is measured in the units of the target/response variable, so it can sometimes be a bit hard to interpret. But in this case, we know that a typical marathon race time is somewhere between 3 - 5 hours. So this model allows us to predict a runner's race time up to about +/-0.6 of an hour, or +/- 36 minutes. This is not *fantastic*, but not *terrible* either. We can certainly use the model to determine roughly whether an athlete will have a bad, good, or excellent race time, but probably cannot reliably distinguish between athletes of a similar caliber.\n",
    "\n",
    "For now, let’s consider this approach to thinking about RMSPE from our testing data set: as long as its not significantly worse than the cross-validation RMSPE of our best model (**Question 8.1**), then we can say that we’re not doing too much worse on the test data than we did on the training data. In future courses on statistical/machine learning, you will learn more about how to interpret RMSPE from testing data and other ways to assess models.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "553011861ab39af59d82c7ff28c78d80",
     "grade": false,
     "grade_id": "cell-ed97bb769cc923e5",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "**Question 8.3** True or False:\n",
    "<br>{points: 1}\n",
    "\n",
    "The RMPSE from our testing data set is *much worse* than the cross-validation RMPSE of our best model. \n",
    "\n",
    "*Assign your answer to an object named `answer8.3`. Make sure your answer is in lowercase and is surrounded by quotation marks (e.g. `\"true\"` or `\"false\"`).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "57ecd2f6ac40fa40cd040ffc75fc7ce5",
     "grade": false,
     "grade_id": "cell-24ccc13552b0fff1",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3b2b852563c34672bc858020794b7a51",
     "grade": true,
     "grade_id": "cell-f39ed1223c189ec6",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "test_8.3()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "35a07e04ba2aae150c37aad410e9c317",
     "grade": false,
     "grade_id": "cell-f527fba0cc1ca89d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "**Question 9.0**\n",
    "<br> {points: 1}\n",
    "\n",
    "Let's visualize what the relationship between `max` and `time_hrs` looks like with our best $k$ value to ultimately explore how the $k$ value affects $k$-nn regression.\n",
    "\n",
    "To do so, use the `predict` function on the workflow analysis that utilizes the best $k$ value (`marathon_best_fit`) to create predictions for the `marathon_training` data. Then, add the column of predictions to the `marathon_training` data frame using `bind_cols`. Name the resulting data frame `marathon_preds`.\n",
    "\n",
    "Next, create a scatterplot with the maximum distance ran per week against the marathon time from `marathon_preds`. Assign your plot to an object called `marathon_plot`. **Plot the predictions as a blue line over the data points.** Remember the fundamentals of effective visualizations such as having a **title** and **human-readable axes**. \n",
    "\n",
    "*Note: use `geom_point` before `geom_line` when creating the plot to ensure tests pass!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dd315f90974d6f742929f677f91a1807",
     "grade": false,
     "grade_id": "cell-e623a65f902a7e98",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "set.seed(2019) # DO NOT CHANGE\n",
    "\n",
    "options(repr.plot.width = 7, repr.plot.height = 7)\n",
    "\n",
    "# your code here\n",
    "fail() # No Answer - remove if you provide an answer\n",
    "marathon_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4a77a1e19f18c0f4a00480edc90d717c",
     "grade": true,
     "grade_id": "cell-5eff9c974a058bdf",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "test_9.0()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d5cf62004c345beddfaf41df95ed2cf3",
     "grade": false,
     "grade_id": "cell-9e4e71e1e8bff87c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "source('cleanup.R')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}