HSML 6295 s1 Software Practice.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### HSML 6295 Session 1 (Course Introduction)\n",
                "\n",
                "#### I. Creating a Data Set\n",
                "\n",
                "Create a sample data set by \n",
                "\n",
                "1. reading in a list of ages for 12 individuals\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "mydata = data.frame(Age = c(21, 32, 37, 43, 47, 51, 55, 61, 64, 66, 68, 72))\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "2. and adding their names.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "attr(mydata, \"row.names\") = c(\" 1 Amy\",\" 2 Ben\",\" 3 Carl\",\" 4 Dan\",\" 5 Ed\", \" 6 Fran\", \" 7 George\", \" 8 Helen\", \" 9 Iris\", \"10 Jen\", \"11 Kim\", \"12 Luke\")\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "Show the data set that we just created:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "mydata\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 1.**\n",
                "What is Carl's age in this data set?\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 2.**\n",
                "Change Amy's age from 21 to 22 in the above code.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 3.**\n",
                "Change the name of the 12th individual from Luke to Lucy.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "The `attach` command tells R that all the variables referred to below are found in the data set `mydata` that we just created:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "attach(mydata)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### II. Summary Statistics 1: Location Measures\n",
                "\n",
                "**Knowledge Check 4.**\n",
                "Consider the 12 values of `Age` in this data set and determine the minimum, mean, median, and maximum values of `Age`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "The `summary` command shows the minimum, first quartile, mean, median, third quartile, and maximum of one or more variables:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "summary(Age)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "We can also represent these numbers graphically in a box plot:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "boxplot(Age)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### III. Summary Statistics 2: Dispersion Measures\n",
                "\n",
                "**Knowledge Check 5.**\n",
                "Compute the variance of `Age` if its mean is\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "round(mean(Age),1)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Recall that the variance is computed as\n",
                "$$\\frac{1}{n-1}\\sum_{i=1}^n (x_i - \\bar{x})^2$$\n",
                "where $x_i$ is the $i$th value of the variable, $\\bar{x}$ is its mean, and $n$ is the number of observations in the data set.\n",
                "\n",
                "**Knowledge Check 6.**\n",
                "Compute the standard deviation of `Age` if its variance is \n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "round(var(Age), 1)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### IV. Plotting the Entire Distribution\n",
                "\n",
                "We can estimate the entire distribution with a histogram:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "hist(Age, cex.main=0.97)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 7.**\n",
                "How many individuals in this data set are between 60 and 70 years old?\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "We can also estimate the density function:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "d = density(Age) # estimate density function\n",
                "plot(d, main=\"Density Function Estimate of Age\", xlab = \"Age\", cex.main=0.97) # plot density function\n",
                "lines(d, type='l', col='red') # color the density function red\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### V. Adding Another Variable\n",
                "\n",
                "Now let's add another variable, `Cost`, which measures annual health care cost for each individual:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "set.seed(8)\n",
                "Cost = 100*Age + Age^2 + round(rnorm(length(Age), mean = 0, sd = 4000))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Note that because the third term is a vector of random numbers, we are using the `set.seed` command to obtain the same `Cost` values every time we run this code.\n",
                "\n",
                "**Knowledge Check 8.**\n",
                "Change the value of the seed from 8 to 4 and recompute the values of `Cost`.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "To ensure that `Cost` is never negative, we set all negative values to zero:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "Cost[Cost < 0] = 0\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "We add this new variable to our existing data set\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "mydata = data.frame(Age, Cost)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "and re-apply the row names.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "attr(mydata, \"row.names\") = c(\"Amy\",\"Ben\",\"Carl\",\"Dan\",\"Ed\", \"Fran\", \"George\", \"Helen\", \"Iris\", \"Jen\", \"Kim\", \"Lucy\")\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "Our data set now looks like this:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "mydata\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 9.**\n",
                "Determine the following:\n",
                "\n",
                "1. Jen's annual health care cost\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "2. The mean annual health care cost\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "3. The median annual health care cost\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "\n",
                "4. The standard deviation of annual health care cost\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "As we did for `Age`, we can also plot the histogram and density function estimate of `Cost`:\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "hist(Cost, cex.main=0.97)\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "d = density(Cost) # estimate density function\n",
                "plot(d, main=\"Density Function Estimate of Cost\", xlab = \"Cost\", cex.main=0.97)  # plot density function\n",
                "lines(d, type='l', col='darkblue')  # color the density function dark blue\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### VI. Scatter Plot\n",
                "\n",
                "We can show both variables, `Age` and `Cost`, in a single graph, called a \"scatter plot\":\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "plot(Age, Cost)\n",
                "with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 10.**\n",
                "On average, does cost increase, decrease, or remain the same as age increases?\n",
                "\n",
                "**Knowledge Check 11.**\n",
                "Is Helen's annual health care cost in the bottom or top half of the cost values of the 12 individuals in this data set? What about Ed's cost?\n",
                "\n",
                "#### VII. Using Age to Predict Cost\n",
                "\n",
                "We can now fit a straight line to the data.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "reg1 = lm(Cost ~ Age, data=subset(mydata, Age < 70))\n",
                "Pred = predict(reg1, mydata)\n",
                "plot(Age, Cost)\n",
                "with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "lines(Age, Pred, type='l', col='blue')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Here we fitted a straight line only to the 11 data points representing the Amy through Kim, who are all younger than 70.\n",
                "The straight line extends to Lucy, however, who is 72.\n",
                "\n",
                "**Knowledge Check 12.**\n",
                "Is the __observed value__ of `Cost` for Lucy larger or smaller than the __predicted value__ of `Cost` for Lucy?\n",
                "\n",
                "A straight line doesn't give us a lot of flexibility to track the data in our sample. We can allow successively more flexible functional forms to track the data, as shown here:\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Model 1\n",
                "  reg = lm(Cost ~ poly(Age, 1), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  plot(Age, Cost, main = paste(\"Model\", 1), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='blue')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Model 2\n",
                "  reg = lm(Cost ~ poly(Age, 2), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  plot(Age, Cost, main = paste(\"Model\", 2), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='red')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Model 3\n",
                "  reg = lm(Cost ~ poly(Age, 4), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  plot(Age, Cost, main = paste(\"Model\", 3), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='coral')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Model 4\n",
                "  reg = lm(Cost ~ poly(Age, 8), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  plot(Age, Cost, main = paste(\"Model\", 4), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='dark green')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 13.**\n",
                "Which model approximates the observed values of the response (`Cost`) in the __training data__ (the 11 `Age`-`Cost` pairs representing Amy through Kim) most closely?\n",
                "\n",
                "**Knowledge Check 14.**\n",
                "Which model approximates the observed value of the response (`Cost`) in the __test data__ (the `Age`-`Cost` pair representing Lucy) most closely?\n",
                "\n",
                "**Knowledge Check 15.**\n",
                "Change the degree of the polynomial, which represents the complexity parameter in this class of models, in Model 4 from 8 to 6.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Rename the title of the figure from \"Model 4\" to \"Model 5\".\n",
                "\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "\n",
                "\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### VIII. Model Tuning\n",
                "\n",
                "Two aggregate measures how closely a model's predicted values $\\hat{y}_i$ approximate the observed values $y_i$:\n",
                "\n",
                "* The __Mean Absolute Error__ (MAE):\n",
                "$$\\frac{1}{n}\\sum_{i=1}^n |y_i - \\hat{y}_i|$$\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Function that returns Mean Absolute Error (MAE)\n",
                "mae = function(error){mean(abs(error))}\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "* The __Root Mean Squared Error__ (RMSE)\n",
                "$$\\sqrt{\\frac{1}{n}\\sum_{i=1}^n (y_i - \\hat{y}_i)^2}$$\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "# Function that returns Root Mean Squared Error (RMSE)\n",
                "rmse = function(error){sqrt(mean(error^2))}\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "In the following 4 figures I have added two measures of the __training error__, namely the values of the RMSE and MAE for the training data.\n",
                "I've also added the __test error__, defined as the absolute value of the difference between the observed and the predicted cost for Lucy.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "  reg = lm(Cost ~ poly(Age, 1), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 1, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='blue')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "  reg = lm(Cost ~ poly(Age, 2), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 2, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='red')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "  reg = lm(Cost ~ poly(Age, 4), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 3, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='coral')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": "\n"
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "  reg = lm(Cost ~ poly(Age, 8), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 4, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='dark green')\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "**Knowledge Check 16.**\n",
                "The following code computes and plots the 4 models above for any value of the seed.\n",
                "You change the value of the seed by changing the number 4 in this line of code\n",
                "\n",
                "`for(i in 4)`\n",
                "\n",
                "to any other integer-valued number.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": null,
            "metadata": {},
            "outputs": [],
            "source": [
                "fn = function(i){set.seed(i)}\n",
                "\n",
                "for(i in 5){\n",
                "  fn(i)\n",
                "  Cost = 100*Age + Age^2 + round(rnorm(length(Age), mean = 0, sd = 4000))\n",
                "  Cost[Cost < 0] = 0\n",
                "  mydata = data.frame(Age, Cost)\n",
                "  attr(mydata, \"row.names\") = c(\"Amy\",\"Ben\",\"Carl\",\"Dan\",\"Ed\", \"Fran\", \"George\", \"Helen\", \"Iris\", \"Jen\", \"Kim\", \"Lucy\")\n",
                "\n",
                "  par(mfrow=c(2,2), oma=c(0,0,2,0))\n",
                "\n",
                "  reg = lm(Cost ~ poly(Age, 1), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 1, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='blue')\n",
                "\n",
                "  reg = lm(Cost ~ poly(Age, 2), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 2, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='red')\n",
                "\n",
                "  reg = lm(Cost ~ poly(Age, 4), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 3, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='coral')\n",
                "\n",
                "  reg = lm(Cost ~ poly(Age, 8), data=subset(mydata, Age < 70))\n",
                "  Pred = predict(reg, mydata)\n",
                "  RMSE = format(round(rmse(reg$residuals)), big.mark=\",\")\n",
                "  MAE  = format(round(mae(reg$residuals)), big.mark=\",\")\n",
                "  test_error = format(round(abs(Cost[12] - Pred[12])), big.mark=\",\")\n",
                "  plot(Age, Cost, main = paste(\"Model\", 4, \"\\nTraining Error:   RMSE \", RMSE, \"  MAE \", MAE, \"\\n Test Error: \", test_error), cex.main=0.97)\n",
                "  with(mydata, text(Cost ~ Age, labels = row.names(mydata), pos = 1))\n",
                "  lines(Age, Pred, type='l', col='dark green')\n",
                "\n",
                "  title(paste(\"Seed Value =\", i), outer=TRUE)\n",
                "}\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "1. Change the seed to every value from 1 through 10.\n",
                "\n",
                "2. Run only the R code in this file using the keyboard shortcut\tCtrl+Shift+S / Command+Shift+S. The plot window in RStudio will show you the results for the 4 models side by side.\n",
                "\n",
                "4. For each value of the seed, record which model achieves the smallest training error as measured by the RMSE, which model achieves the smallest training error as measured by the MAE, and which model achieves the smallest test error. Record your results in the following table by putting the seed value in the corresponding cell.\n",
                "\n",
                "For instance, when the seed value is 5, Model 4 minimizes both kinds of training error and Model 1 achieves the smallest test error.\n",
                "\n",
                "Performance Metric | Model 1 | Model 2 | Model 3 | Model 4\n",
                "--- | ---: | ---: | ---: | ---:\n",
                "Training Error (RMSE) | | | | 5\n",
                "Training Error (MAE) | | | | 5\n",
                "Test Error | 5 | | |\n",
                "\n",
                "5. For each performance metric (Training Error (RMSE), Training Error (MAE), Test Error), count how often each model was the top performer.\n",
                "\n",
                "Performance Metric | Model 1 | Model 2 | Model 3 | Model 4\n",
                "--- | ---: | ---: | ---: | ---:\n",
                "Training Error (RMSE) | | | | \n",
                "Training Error (MAE) | | | |\n",
                "Test Error | | | |\n",
                "\n",
                "6. Which model would you choose to predict cost based on age?\n"
            ]
        }
    ],
    "metadata": {
        "anaconda-cloud": "",
        "kernelspec": {
            "display_name": "R",
            "langauge": "R",
            "name": "ir"
        },
        "language_info": {
            "codemirror_mode": "r",
            "file_extension": ".r",
            "mimetype": "text/x-r-source",
            "name": "R",
            "pygments_lexer": "r",
            "version": "3.4.1"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}