Part-4-Resampling-Grid-Search.html

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
  <head>
    <title>Applied Machine Learning - Resampling and Grid Search</title>
    <meta charset="utf-8" />
    <meta name="author" content="Max Kuhn (RStudio)" />
    <script src="libs/htmlwidgets-1.5.1/htmlwidgets.js"></script>
    <script src="libs/jquery-1.12.4/jquery.min.js"></script>
    <link href="libs/leaflet-1.3.1/leaflet.css" rel="stylesheet" />
    <script src="libs/leaflet-1.3.1/leaflet.js"></script>
    <link href="libs/leafletfix-1.0.0/leafletfix.css" rel="stylesheet" />
    <script src="libs/Proj4Leaflet-1.0.1/proj4-compressed.js"></script>
    <script src="libs/Proj4Leaflet-1.0.1/proj4leaflet.js"></script>
    <link href="libs/rstudio_leaflet-1.3.1/rstudio_leaflet.css" rel="stylesheet" />
    <script src="libs/leaflet-binding-2.0.2/leaflet.js"></script>
    <script src="libs/leaflet-providers-1.1.17/leaflet-providers.js"></script>
    <script src="libs/leaflet-providers-plugin-2.0.2/leaflet-providers-plugin.js"></script>
    <link href="libs/countdown-0.3.3/countdown.css" rel="stylesheet" />
    <script src="libs/countdown-0.3.3/countdown.js"></script>
    <link rel="stylesheet" href="mtheme_max.css" type="text/css" />
    <link rel="stylesheet" href="fonts_mtheme_max.css" type="text/css" />
  </head>
  <body>
    <textarea id="source">
class: center, middle, inverse, title-slide

# Applied Machine Learning - Resampling and Grid Search
### Max Kuhn (RStudio)

---


# Loading


```r
library(tidymodels)
```

```
## Registered S3 method overwritten by 'xts':
##   method     from
##   as.zoo.xts zoo
```

```
## ── Attaching packages ──────────────────────────────── tidymodels 0.0.2 ──
```

```
## ✔ broom     0.5.1          ✔ purrr     0.3.3     
## ✔ dials     0.0.3.9002     ✔ recipes   0.1.7.9001
## ✔ dplyr     0.8.3          ✔ rsample   0.0.5     
## ✔ ggplot2   3.2.1          ✔ tibble    2.1.3     
## ✔ infer     0.4.0          ✔ yardstick 0.0.4     
## ✔ parsnip   0.0.4
```

```
## ── Conflicts ─────────────────────────────────── tidymodels_conflicts() ──
## ✖ purrr::discard()  masks scales::discard()
## ✖ dplyr::filter()   masks stats::filter()
## ✖ dplyr::lag()      masks stats::lag()
## ✖ ggplot2::margin() masks dials::margin()
## ✖ dials::offset()   masks stats::offset()
## ✖ recipes::step()   masks stats::step()
```


---
# Previously


.pull-left[

.code70[


```r
library(AmesHousing)

ames &lt;- 
  make_ames() %&gt;% 
  dplyr::select(-matches("Qu"))

set.seed(4595)
data_split &lt;- initial_split(ames, strata = "Sale_Price")
ames_train &lt;- training(data_split)
ames_test  &lt;- testing(data_split)

perf_metrics &lt;- metric_set(rmse, rsq, ccc)
```

]

]
.pull-right[

.code70[


```r
ames_rec &lt;- 
  recipe(Sale_Price ~ Bldg_Type + Neighborhood + Year_Built + 
           Gr_Liv_Area + Full_Bath + Year_Sold + Lot_Area +
           Central_Air + Longitude + Latitude,
         data = ames_train) %&gt;%
  step_log(Sale_Price, base = 10) %&gt;%
  step_BoxCox(Lot_Area, Gr_Liv_Area) %&gt;%
  step_other(Neighborhood, threshold = 0.05)  %&gt;%
  step_dummy(all_nominal()) %&gt;%
  step_interact(~ starts_with("Central_Air"):Year_Built) %&gt;%
  step_bs(Longitude, Latitude, deg_free = 5)
```

]

]


---
layout: false
class: inverse, middle, center

#  Resampling

---

# Resampling Methods

.pull-left[
These are additional data splitting schemes that are applied to the _training_ set and are used for **estimating model performance**. 

They attempt to simulate slightly different versions of the training set. These versions of the original are split into two model subsets:

* The _analysis set_ is used to fit the model (analogous to the training set). 
* Performance is determined using the _assessment set_. 

This process is repeated many times. 

There are [different flavors of resampling](https://bookdown.org/max/FES/resampling.html) but we will focus on one method in these notes.

]

.pull-right[

&lt;img src="images/resampling.svg" width="120%" style="display: block; margin: auto;" /&gt;

]


---

# The Model Workflow and Resampling

All resampling methods repeat this process multiple times: 

&lt;br&gt; 

&lt;img src="images/diagram-resampling.svg" width="65%" style="display: block; margin: auto;" /&gt;

&lt;br&gt; 

The final resampling estimate is the average of all of the estimated metrics (e.g. RMSE, etc).


---

# V-Fold Cross-Validation

.pull-left[

Here, we randomly split the training data into _V_ distinct blocks of roughly equal size (AKA the "folds").

* We leave out the first block of analysis data and fit a model.

* This model is used to predict the held-out block of assessment data.

* We continue this process until we've predicted all _V_ assessment blocks

The final performance is based on the hold-out predictions by _averaging_ the statistics from the _V_ blocks. 

]

.pull-right[

&lt;img src="images/part-4-rs-diagram-1.svg" width="95%" style="display: block; margin: auto;" /&gt;

&lt;br&gt;

_V_ is usually taken to be 5 or 10 and leave-one-out cross-validation has each sample as a block. 

**Repeated CV** can be used when trianing set sizes are small. 5 repeats of 10-fold CV averages 50 sets of metrics. 

]

---

#  10-Fold Cross-Validation with _n_ = 50

&lt;img src="images/part-4-cv-plot-1.svg" width="60%" style="display: block; margin: auto;" /&gt;

---

# Resampling Results

The goal of resampling is to produce a single estimate of perforamnce for a model. 

Even though we end up estimating _V_ models (for _V_-fold CV), these models are discarded after we have our performance estimate. 

Resampling is basically an _emprical simulation system_ used to understand how well the model would work on _new data_.


---

# Cross-Validating Using `rsample` &lt;img src="images/recipes.png" class="title-hex"&gt;

.pull-left[


```r
set.seed(2453)
cv_splits &lt;- vfold_cv(ames_train) #10-fold is default
cv_splits
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 2
##    splits           id    
##    &lt;named list&gt;     &lt;chr&gt; 
##  1 &lt;split [2K/220]&gt; Fold01
##  2 &lt;split [2K/220]&gt; Fold02
##  3 &lt;split [2K/220]&gt; Fold03
##  4 &lt;split [2K/220]&gt; Fold04
##  5 &lt;split [2K/220]&gt; Fold05
##  6 &lt;split [2K/220]&gt; Fold06
##  7 &lt;split [2K/220]&gt; Fold07
##  8 &lt;split [2K/220]&gt; Fold08
##  9 &lt;split [2K/220]&gt; Fold09
## 10 &lt;split [2K/219]&gt; Fold10
```

]

.pull-right[

Each individual split object is similar to the 
`initial_split()` example.


```r
cv_splits$splits[[1]]
```

```
## &lt;1979/220/2199&gt;
```

```r
cv_splits$splits[[1]] %&gt;% analysis() %&gt;% dim()
```

```
## [1] 1979   74
```

```r
cv_splits$splits[[1]] %&gt;% assessment() %&gt;% dim()
```

```
## [1] 220  74
```

]

???

Note that `&lt;split [2K/222]&gt;` rounds to the thousandth and is the same as `&lt;1977/222/2199&gt;`


---

# K-Nearest Neighbors Model

_K_-nearest neighbors stores the training set (including the outcome). 

When a new sample is predicted, _K_ training set points are found that are most similar to the new sample being predicted. 

The predicted value for the new sample is some summary statistic of the neighbors, usually: 

* the mean for regression, or
* the mode for classification.

Let's try a 5-neighbor model on the Ames data. 

---

# _5_-Nearest Neighbors Model

<div id="htmlwidget-d8037c81783ee8105465" style="width:80%;height:504px;" class="leaflet html-widget"></div>
<script type="application/json" data-for="htmlwidget-d8037c81783ee8105465">{"x":{"options":{"crs":{"crsClass":"L.CRS.EPSG3857","code":null,"proj4def":null,"projectedBounds":null,"options":{}}},"setView":[[42.060936,-93.641325],18,[]],"calls":[{"method":"addProviderTiles","args":["CartoDB.DarkMatterNoLabels",null,null,{"errorTileUrl":"","noWrap":false,"detectRetina":false}]},{"method":"addCircles","args":[[42.060977,42.061114,42.061149,42.061318,42.061193],[-93.641487,-93.641488,-93.641772,-93.641781,-93.642447],6,null,null,{"interactive":true,"className":"","stroke":true,"color":"red","weight":5,"opacity":0.4,"fill":true,"fillColor":"red","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addCircles","args":[42.060936,-93.641325,6,null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":5,"opacity":0.4,"fill":true,"fillColor":"yellow","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addCircles","args":[[42.052659,42.051245,42.060899,42.060779,42.062978,42.060728,42.06112,42.059193,42.05848,42.058151,42.057268,42.059169,42.061239,42.0603514,42.056673,42.054453,42.05027,42.049297,42.054592,42.055227,42.053395,42.056375,42.055318,42.051835,42.051685,42.049672,42.050683,42.050681,42.049811,42.062352,42.062109,42.0611331,42.060872,42.060879,42.059617,42.058599,42.057102,42.057164,42.058252,42.057181,42.061928,42.06257,42.062303,42.054386,42.053243,42.051721,42.051674,42.052437,42.050909,42.05306,42.051653,42.050084,42.051429,42.036113,42.036026,42.036659,42.034589,42.0347,42.036633,42.034555,42.034678,42.036535,42.034793,42.034856,42.03457,42.034603,42.046825,42.045773,42.047452,42.048086,42.048566,42.047899,42.047876,42.047782,42.0471748,42.0470316,42.04638,42.043704,42.034561,42.048606,42.046322,42.043306,42.043329,42.043822,42.043821,42.046574,42.042205,42.043363,42.039085,42.038971,42.040391,42.0418387,42.039211,42.039211,42.038538,42.035256,42.035482,42.035943,42.036095,42.037424,42.047447,42.043984,42.042908,42.0430839,42.043969,42.044052,42.0435919,42.042308,42.040396,42.04137,42.038515,42.039841,42.038443,42.038521,42.0381282,42.035869,42.0372097,42.040085,42.041119,42.035867,42.03594,42.0347571,42.034848,42.034748,42.034851,42.036873,42.037093,42.033313,42.031363,42.031381,42.0303783,42.033644,42.03128,42.0303113,42.030383,42.030467,42.030388,42.030331,42.029146,42.028145,42.028164,42.02916,42.028165,42.033642,42.033621,42.0336,42.033665,42.033631,42.032302,42.027986,42.028888,42.028875,42.026914,42.020693,42.024141,42.026741,42.033454,42.028256,42.026752,42.02616,42.025255,42.034451,42.033386,42.033387,42.033008,42.03237,42.029867,42.03069,42.029924,42.029731,42.030519,42.03136,42.03128,42.030082,42.03009,42.03387,42.026547,42.024673,42.023079,42.028226,42.026533,42.024208,42.025173,42.0247497,42.025019,42.024162,42.032076,42.033783,42.03068,42.025747,42.024604,42.024604,42.030601,42.0298532,42.024318,42.021321,42.01867,42.022186,42.022045,42.021395,42.021133,42.018662,42.019069,42.016121,42.018721,42.017749,42.016275,42.016253,42.01404,42.014023,42.0203585,42.021245,42.020635,42.021163,42.020306,42.019849,42.019629,42.020196,42.021793,42.022674,42.022702,42.0177987,42.019091,42.019093,42.014937,42.014682,42.011005,42.010606,42.009337,42.010797,42.022772,42.021414,42.021474,42.021456,42.021327,42.02152,42.008687,41.99706,41.997052,41.996065,41.994013,41.997741,41.995346,41.994284,41.99353,41.995489,41.99435,41.992989,41.99321,41.990934,41.991861,41.991758,41.991876,41.992212,41.9917,41.991824,41.991558,41.988729,41.9889358,41.986647,41.9913,41.989848,41.989185,42.05139,42.059956,42.060896,42.060598,42.061178,42.058274,42.058059,42.059645,42.058194,42.05793,42.058173,42.057854,42.058747,42.059242,42.059817,42.058439,42.055408,42.05184,42.052507,42.052102,42.052719,42.05288,42.050345,42.049394,42.051619,42.0503771,42.049406,42.053166,42.052448,42.05267,42.04891,42.056398,42.054626,42.055184,42.05337,42.053306,42.053307,42.0564419,42.056461,42.052758,42.052338,42.051661,42.051823,42.051822,42.050705,42.050682,42.050533,42.050213,42.05009,42.050134,42.050222,42.0488934,42.048486,42.0623099,42.063304,42.063025,42.061902,42.062991,42.062992,42.062994,42.062942,42.062258,42.062776,42.062789,42.0623985,42.062108,42.060934,42.060353,42.061382,42.061437,42.060576,42.061678,42.060178,42.0599123,42.0598989,42.061404,42.059241,42.058886,42.0592,42.059177,42.059179,42.058336,42.057177,42.057089,42.057019,42.056983,42.057467,42.06232,42.062562,42.062305,42.062081,42.062086,42.061562,42.059377,42.061739,42.061656,42.061365,42.063283,42.058368,42.05859,42.059481,42.059068,42.057839,42.057834,42.057833,42.057093,42.056955,42.058668,42.053749,42.053911,42.055626,42.053441,42.053041,42.052668,42.050411,42.049701,42.05161,42.050215,42.049921,42.050971,42.054282,42.055036,42.054302,42.054152,42.053258,42.053108,42.051797,42.051645,42.051648,42.051652,42.050406,42.052346,42.052127,42.05142,42.051378,42.051251,42.037611,42.037661,42.036011,42.035282,42.038127,42.037756,42.037434,42.036852,42.036905,42.036576,42.035518,42.037607,42.035968,42.0365385,42.035287,42.035806,42.034747,42.035289,42.035331,42.034595,42.034581,42.037236,42.048416,42.048488,42.047338,42.047253,42.047101,42.046418,42.047171,42.043782,42.034509,42.047708,42.04896,42.04876,42.047652,42.046476,42.045806,42.045799,42.043236,42.043842,42.043673,42.044693,42.048796,42.0467129,42.046635,42.044821,42.043935,42.043135,42.044843,42.044774,42.04488,42.038492,42.0407264,42.040011,42.040888,42.041104,42.039463,42.038596,42.035489,42.03464,42.035951,42.036178,42.048294,42.046827,42.048608,42.04806,42.04579,42.044886,42.043966,42.043991,42.04297,42.042233,42.040931,42.040857,42.0404571,42.039674,42.0385265,42.038356,42.038398,42.037122,42.034918,42.034686,42.034754,42.036223,42.035829,42.034746,42.0380659,42.037679,42.037144,42.037328,42.036015,42.036002,42.034574,42.035257,42.0417879,42.04015,42.040295,42.04124,42.039047,42.040766,42.038811,42.037129,42.035712,42.035787,42.035841,42.037233,42.03602,42.035968,42.035706,42.037214,42.034858,42.037049,42.038948,42.038257,42.035287,42.0342224,42.0320748,42.0323259,42.032396,42.032202,42.032108,42.0319053,42.031305,42.0304742,42.0303538,42.030282,42.033381,42.033579,42.032048,42.032301,42.032227,42.030523,42.030196,42.0290352,42.028957,42.029305,42.028104,42.028556,42.0282044,42.027062,42.027227,42.023218,42.033558,42.0333765,42.031564,42.031542,42.033406,42.033616,42.03329,42.033347,42.03252,42.03246,42.031292,42.032375,42.03235,42.0310861,42.030062,42.028948,42.027854,42.027998,42.027039,42.026956,42.025306,42.023085,42.025796,42.024143,42.022828,42.032681,42.028101,42.028098,42.028049,42.0269586,42.022825,42.033633,42.032388,42.029863,42.030628,42.031138,42.033427,42.032302,42.032652,42.032564,42.032503,42.023229,42.028468,42.027656,42.027506,42.024483,42.023982,42.023916,42.026133,42.024993,42.0246851,42.023548,42.033714,42.0318449,42.03254,42.030625,42.030678,42.03252,42.031622,42.031834,42.031623,42.031545,42.0314669,42.0314606,42.0312822,42.0310946,42.0309301,42.026626,42.026482,42.026414,42.026372,42.025707,42.025293,42.024628,42.024613,42.024765,42.024327,42.024399,42.029772,42.0255464,42.021171,42.021176,42.021207,42.020817,42.018996,42.0197228,42.018597,42.017569,42.016903,42.018667,42.016862,42.016399,42.01571,42.016551,42.01613,42.015713,42.021384,42.020087,42.019697,42.019691,42.020666,42.020991,42.021788,42.020843,42.017109,42.018722,42.018847,42.01877,42.018703,42.016973,42.016978,42.0181479,42.018875,42.017856,42.016819,42.016797,42.016274,42.016271,42.016468,42.013957,42.013512,42.020123,42.021219,42.018988,42.0196717,42.017026,42.01896,42.019423,42.020321,42.0217997,42.021496,42.019004,42.019381,42.018592,42.022092,42.022091,42.020123,42.019713,42.017564,42.016921,42.016182,42.018931,42.017889,42.0189256,42.016835,42.016957,42.016067,42.0191252,42.019181,42.018707,42.017574,42.016684,42.0158534,42.013169,42.014943,42.013753,42.009489,42.009489,42.009489,42.009489,42.010636,42.01141,42.021425,42.021596,42.020184,42.008683,42.0183,41.996672,41.997119,41.99539,41.997067,41.997069,41.996919,41.996066,42.00138,41.995027,41.995058,41.997684,41.998952,41.9981656,41.995359,41.992964,41.9935397,41.992193,41.993128,41.992122,41.991786,41.992919,41.992185,41.9928219,41.99217,41.992539,41.991786,41.991365,41.987535,41.988776,41.988222,41.986509,41.991493,41.99095,42.0525525,42.051446,42.05996,42.061231,42.060771,42.060822,42.060369,42.060639,42.062976,42.061374,42.06076,42.059364,42.059411,42.058445,42.05701,42.059518,42.061394,42.060241,42.059792,42.0586718,42.056015,42.054514,42.054562,42.054439,42.055358,42.054503,42.054211,42.055432,42.052544,42.051473,42.050679,42.049449,42.04986,42.051074,42.056306,42.05539,42.054328,42.0564348,42.052794,42.052755,42.051479,42.052332,42.051678,42.051687,42.051659,42.050001,42.050257,42.04845,42.062046,42.0632692,42.063129,42.062259,42.0624143,42.061133,42.06027,42.060427,42.060425,42.0599011,42.058484,42.058683,42.058573,42.058649,42.057028,42.058694,42.057164,42.0579589,42.056766,42.057446,42.0571198,42.057468,42.057094,42.056616,42.062299,42.062285,42.062004,42.061544,42.061312,42.058377,42.057986,42.058595,42.059507,42.05695,42.058581,42.053879,42.053666,42.055748,42.055777,42.0522319,42.052272,42.050383,42.05162,42.049501,42.054236,42.055102,42.054366,42.054306,42.054319,42.055517,42.054227,42.053183,42.051646,42.05165,42.050307,42.052278,42.052456,42.0520487,42.05138,42.052173,42.051224,42.050899,42.037836,42.035271,42.035132,42.037918,42.037806,42.03805,42.036525,42.035875,42.035878,42.034988,42.036089,42.034535,42.036501,42.036466,42.036039,42.035179,42.034614,42.035703,42.03459,42.0375,42.034532,42.047666,42.048607,42.047467,42.048089,42.048191,42.047806,42.047571,42.047102,42.047103,42.046381,42.046377,42.046372,42.046085,42.046961,42.046962,42.046647,42.047171,42.047171,42.047171,42.045012,42.044642,42.036729,42.047928,42.049141,42.048762,42.046399,42.04378,42.0471033,42.046584,42.048185,42.044262,42.044058,42.044851,42.043845,42.044101,42.044773,42.044773,42.04066,42.040112,42.0417754,42.0417764,42.041811,42.038323,42.04102,42.0418699,42.041172,42.035571,42.034563,42.035741,42.0380577,42.036178,42.036102,42.03713,42.034679,42.048952,42.047782,42.0470858,42.0470799,42.046935,42.046572,42.044736,42.044791,42.044853,42.042187,42.042023,42.042304,42.040479,42.040301,42.040391,42.039531,42.0372039,42.03471,42.037302,42.035893,42.035723,42.035222,42.040021,42.040142,42.041173,42.038686,42.037104,42.035813,42.037162,42.037006,42.037291,42.034564,42.034789,42.034686,42.0347301,42.034636,42.036973,42.037025,42.0342174,42.034128,42.033379,42.032611,42.032419,42.030427,42.0304142,42.030418,42.030482,42.032161,42.032272,42.030406,42.0303597,42.033249,42.032008,42.032094,42.032381,42.0311,42.0304823,42.029028,42.02928,42.029281,42.028287,42.02793,42.027073,42.027032,42.0288284,42.028397,42.029209,42.0273991,42.0272517,42.02774,42.026691,42.03361,42.0343116,42.03338,42.03047,42.033456,42.033568,42.03337,42.033511,42.033321,42.033473,42.033366,42.033493,42.031354,42.031293,42.031493,42.031461,42.031496,42.031467,42.031432,42.031049,42.030388,42.031057,42.029399,42.028426,42.02903,42.029038,42.0255246,42.024111,42.0249938,42.0242588,42.022574,42.02295,42.026478,42.024885,42.0242331,42.034409,42.033423,42.02393,42.033604,42.033225,42.032365,42.031571,42.031248,42.031123,42.031348,42.03156,42.030552,42.034428,42.034428,42.033454,42.032495,42.032566,42.032155,42.032653,42.029453,42.024662,42.0243033,42.02521,42.028568,42.028595,42.027332,42.027507,42.028381,42.028188,42.0280297,42.025036,42.0236702,42.0244599,42.022902,42.022769,42.02283,42.031766,42.033912,42.03085,42.031077,42.030621,42.025411,42.027368,42.027193,42.0249247,42.025781,42.021173,42.01915,42.019153,42.018501,42.017423,42.017517,42.017914,42.017766,42.016276,42.01626,42.015949,42.022136,42.021234,42.019769,42.020694,42.02107,42.021275,42.020796,42.018412,42.018709,42.017598,42.018006,42.01727,42.016207,42.014435,42.016709,42.017866,42.016254,42.017854,42.016262,42.013952,42.013912,42.013641,42.013622,42.015927,42.015927,42.016121,42.016122,42.01975,42.019937,42.021244,42.019997,42.019996,42.02099,42.019861,42.018991,42.018852,42.01883,42.018898,42.0202945,42.020394,42.019544,42.020112,42.021582,42.01797,42.017962,42.019959,42.018541,42.017874,42.02224,42.022238,42.0214496,42.019986,42.017413,42.017083,42.019272,42.019092,42.019005,42.01791,42.017834,42.0176853,42.01789,42.0168678,42.016072,42.019833,42.018443,42.018888,42.018766,42.016769,42.015667,42.014567,42.012638,42.012488,42.009489,42.009489,42.022607,42.021806,42.022516,42.020379,42.021389,42.018806,42.021162,42.021594,42.022626,42.009326,42.008803,42.008806,42.0091899,41.99507,41.997124,41.997062,41.997069,41.996903,42.00514,41.998999,41.994841,41.99373,41.995142,41.993751,41.995175,41.999812,41.999575,41.9976055,41.995512,41.99639,41.99423,41.993611,41.99338,41.990054,41.993059,41.993185,41.99261,41.991372,41.991883,41.991779,41.992133,41.991933,41.988675,41.987109,41.988934,41.98811,41.987289,41.98658,41.991885,42.022745,42.051297,42.051357,42.049671,42.050443,42.060466,42.061664,42.061326,42.063067,42.059592,42.059517,42.059242,42.059293,42.05817,42.057533,42.057022,42.059375,42.059309,42.060431,42.060144,42.060096,42.059743,42.058998,42.058996,42.054368,42.053916,42.055545,42.0545,42.054941,42.055217,42.054427,42.05622,42.052353,42.049832,42.050075,42.050372,42.049956,42.051039,42.048765,42.055264,42.053407,42.054612,42.053547,42.0562952,42.056395,42.05467,42.054789,42.055319,42.052669,42.052698,42.052696,42.052294,42.051833,42.05057,42.0488205,42.062574,42.063296,42.0632722,42.063141,42.0619544,42.061976,42.0619709,42.061284,42.062259,42.0628153,42.0624082,42.062434,42.061504,42.06027,42.060269,42.060576,42.060427,42.059561,42.06113,42.059677,42.058634,42.058619,42.059211,42.059202,42.059197,42.059194,42.059175,42.059176,42.059176,42.05826,42.058276,42.058284,42.056827,42.057124,42.05738,42.0568529,42.057209,42.063388,42.062483,42.062279,42.062887,42.062931,42.062974,42.063019,42.063106,42.063342,42.062584,42.062581,42.062946,42.063301,42.061713,42.061429,42.061483,42.061541,42.061531,42.061392,42.060021,42.059181,42.057981,42.058401,42.059526,42.0533209,42.055751,42.05359,42.053467,42.05198,42.052361,42.052247,42.049509,42.054408,42.055757,42.054318,42.054177,42.054417,42.054354,42.054154,42.054153,42.054153,42.054157,42.053185,42.053108,42.053032,42.053067,42.051802,42.051655,42.050206,42.050301,42.052269,42.051377,42.052267,42.052146,42.051054,42.037586,42.037643,42.035259,42.036038,42.036464,42.034569,42.035678,42.0362103,42.034678,42.035871,42.035857,42.034532,42.034724,42.035675,42.034652,42.034819,42.037502,42.034644,42.047384,42.045776,42.047618,42.0474249,42.047406,42.047705,42.047699,42.047687,42.047667,42.047104,42.047104,42.046144,42.047171,42.046137,42.046135,42.046411,42.044383,42.044989,42.042994,42.043754,42.043048,42.042961,42.037103,42.0349702,42.046549,42.048067,42.04849,42.047615,42.043363,42.043005,42.04259,42.04389,42.046504,42.043121,42.044036,42.042485,42.042479,42.044921,42.043609,42.039141,42.040143,42.0417369,42.0413937,42.04074,42.041245,42.041332,42.039212,42.0392,42.035684,42.036023,42.0362,42.03465,42.034906,42.049216,42.048934,42.045809,42.0459325,42.046882,42.044053,42.044735,42.043932,42.043946,42.044218,42.043952,42.043233,42.0426969,42.0418697,42.041243,42.040451,42.03981,42.039369,42.03846,42.0384781,42.034789,42.0347206,42.034812,42.036077,42.036027,42.03725,42.0380982,42.0357,42.035779,42.035854,42.034878,42.039338,42.039983,42.040146,42.038888,42.035733,42.035921,42.035876,42.037106,42.035862,42.036002,42.037084,42.034685,42.034614,42.036949,42.035423,42.0346,42.035034,42.0314733,42.03131,42.031369,42.0311108,42.033405,42.032343,42.033385,42.033489,42.033313,42.032276,42.030579,42.030311,42.03013,42.0299846,42.029218,42.029198,42.029223,42.0280396,42.028076,42.02819,42.027154,42.0282,42.0277574,42.026089,42.0268089,42.0272605,42.0272703,42.033311,42.034339,42.031846,42.030823,42.033482,42.033612,42.033336,42.032399,42.031433,42.031438,42.032386,42.031321,42.031483,42.0314873,42.0304477,42.029451,42.02752,42.0275775,42.025229,42.021576,42.0249982,42.0227,42.0238856,42.022963,42.0238391,42.026723,42.0261614,42.022951,42.033524,42.033285,42.028048,42.028004,42.033272,42.032426,42.034429,42.033193,42.032385,42.032098,42.031533,42.031215,42.029679,42.031139,42.029786,42.034422,42.032528,42.034421,42.033557,42.032638,42.033726,42.034018,42.02868,42.026185,42.023229,42.023021,42.028424,42.028236,42.024949,42.025818,42.0205201,42.020186,42.0246868,42.023874,42.023945,42.023451,42.022824,42.034453,42.033662,42.033576,42.032467,42.031566,42.030801,42.02667,42.026642,42.025666,42.025707,42.025594,42.025264,42.024598,42.024594,42.024633,42.025726,42.025156,42.024635,42.024485,42.029803,42.027956,42.027969,42.024305,42.022228,42.021324,42.020966,42.020912,42.019083,42.019028,42.018378,42.017907,42.016896,42.016829,42.016297,42.016214,42.021992,42.021386,42.022239,42.01972,42.021163,42.021056,42.021025,42.020861,42.021737,42.018518,42.018424,42.018849,42.016973,42.016961,42.016212,42.0172,42.0185298,42.018017,42.017856,42.017753,42.015974,42.014823,42.013576,42.013544,42.01343,42.013467,42.01469,42.0210331,42.020703,42.021221,42.021195,42.02103,42.020893,42.019926,42.0196959,42.019902,42.01917,42.018873,42.0164532,42.016642,42.018965,42.020062,42.020783,42.020504,42.022655,42.021436,42.017962,42.018943,42.01873,42.019937,42.018039,42.017106,42.0162813,42.016216,42.016217,42.016217,42.016853,42.019952,42.018796,42.020037,42.018504,42.018941,42.017573,42.016961,42.01629,42.0145503,42.012488,42.009321,42.011343,42.010333,42.010076,42.022836,42.022474,42.022658,42.022566,42.022685,42.0228,42.021547,42.021278,42.021497,42.021555,42.009401,42.009401,41.994357,41.994207,41.994206,41.995218,41.995973,41.995512,42.001694,41.994241,41.994137,41.9941,41.994824,41.995088,41.997992,41.997364,41.997126,41.996089,41.995508,41.997299,41.995341,41.993986,41.997352,41.993288,41.992975,41.993356,41.9927922,41.991211,41.992917,41.99284,41.991709,41.991641,41.991868,41.991612,41.991098,41.989581,41.990264,41.989905,41.987606,41.989768,41.98822,41.986588,42.053327,42.052519,42.050507,42.059909,42.06048,42.060736,42.062828,42.06281,42.062802,42.062956,42.060625,42.0591,42.058421,42.058201,42.059599,42.059332,42.058136,42.060527,42.060516,42.061478,42.061008,42.059957,42.061096,42.0593867,42.056702,42.058563,42.058816,42.055607,42.054367,42.0543376,42.054956,42.0562619,42.05327,42.0524029,42.053418,42.049771,42.052522,42.052359,42.04977,42.049608,42.055131,42.053459,42.054385,42.055319,42.051798,42.051839,42.051673,42.051664,42.048947,42.050705,42.061472,42.063307,42.061901,42.0628074,42.062109,42.061134,42.060342,42.060866,42.060424,42.060357,42.060377,42.061544,42.061377,42.058544,42.05869,42.059216,42.059017,42.059184,42.057164,42.058238,42.057164,42.057164,42.05758,42.062361,42.061915,42.061987,42.063329,42.063185,42.062374,42.0623,42.063301,42.063301,42.063301,42.062831,42.062873,42.060239,42.060293,42.059803,42.060074,42.05865,42.05853,42.057905,42.057838,42.05696,42.058461,42.053314,42.055601,42.053466,42.05178,42.049423,42.051073,42.049687,42.049528,42.055056,42.054176,42.054175,42.053183,42.053033,42.0518,42.052265,42.052272,42.052346,42.051355,42.051426,42.051382,42.051245,42.051225,42.051224,42.037843,42.036149,42.037915,42.037911,42.037827,42.036605,42.036849,42.03674,42.035629,42.03748,42.035235,42.036087,42.036206,42.035302,42.035094,42.035696,42.035841,42.035325,42.03457,42.03458,42.034663,42.037065,42.045763,42.048608,42.047997,42.048058,42.047592,42.04768,42.047665,42.047642,42.047618,42.047595,42.046145,42.046429,42.044609,42.042369,42.042826,42.043351,42.037409,42.045632,42.048199,42.048997,42.046472,42.04806,42.044669,42.046496,42.045706,42.045728,42.045728,42.045885,42.045878,42.04795,42.04657,42.045685,42.044308,42.043214,42.044978,42.042114,42.040072,42.038434,42.04102,42.0414084,42.041314,42.040273,42.040143,42.039202,42.035836,42.035379,42.0380632,42.034612,42.049031,42.048528,42.0422973,42.042156,42.043129,42.044735,42.04498,42.044975,42.044254,42.0427,42.042905,42.041821,42.040252,42.039387,42.038602,42.0383475,42.038836,42.040276,42.039351,42.034934,42.034617,42.0378602,42.037001,42.037097,42.037041,42.0367836,42.035857,42.034574,42.035277,42.034761,42.040944,42.040811,42.040105,42.040081,42.040297,42.040148,42.0385,42.041874,42.03902,42.036098,42.035996,42.036099,42.036053,42.034805,42.03471,42.034662,42.034829,42.034757,42.034708,42.036792,42.03548,42.034753,42.0345614,42.0334994,42.032346,42.03139,42.031506,42.031347,42.030396,42.031196,42.033623,42.032305,42.030267,42.032064,42.032267,42.031964,42.030368,42.03146,42.031138,42.029136,42.029281,42.0281148,42.027953,42.028963,42.029025,42.027992,42.0272703,42.03179,42.033306,42.033369,42.032299,42.031499,42.032396,42.031415,42.031436,42.032417,42.0310818,42.030389,42.028544,42.026886,42.02689,42.025258,42.025459,42.0249549,42.0242386,42.023117,42.0238089,42.034576,42.033296,42.0331453,42.0277962,42.028002,42.027997,42.033521,42.0333,42.032383,42.031356,42.032216,42.030702,42.031362,42.031347,42.030231,42.030232,42.033285,42.032291,42.032307,42.033686,42.032478,42.031635,42.032157,42.032575,42.029453,42.028836,42.023651,42.024705,42.02653,42.027664,42.026614,42.02821,42.028191,42.026458,42.025102,42.025197,42.026308,42.0247523,42.022824,42.032082,42.032493,42.030933,42.024601,42.024555,42.024615,42.025218,42.025194,42.02437,42.023306,42.023308,42.029884,42.028168,42.025573,42.02438,42.024592,42.022227,42.021209,42.0209,42.018123,42.017631,42.01686,42.016217,42.016913,42.020145,42.02007,42.021811,42.021889,42.021785,42.01834,42.018664,42.017055,42.018188,42.016979,42.016904,42.01869,42.017856,42.017867,42.017858,42.016252,42.016618,42.015914,42.016092,42.014636,42.014109,42.015793,42.016004,42.01397,42.015765,42.013519,42.014593,42.0195855,42.020995,42.020342,42.020592,42.020547,42.01899,42.021168,42.018648,42.018961,42.01896,42.018959,42.018813,42.018812,42.01881,42.018809,42.02011,42.022676,42.022659,42.021432,42.018848,42.017727,42.022089,42.021559,42.020871,42.020286,42.018837,42.018282,42.01924,42.01895,42.018808,42.018857,42.016941,42.0165794,42.016072,42.017609,42.01874,42.019759,42.017131,42.016827,42.016385,42.017589,42.0157866,42.016124,42.013296,42.013495,42.01359,42.011793,42.022786,42.022709,42.020441,42.020855,42.020207,42.02153,42.019025,42.019349,42.019099,42.022603,42.009398,42.008375,41.996849,41.994357,41.995987,41.996907,41.995702,41.996063,42.004509,41.997906,41.999553,41.998023,41.999342,41.994347,41.994229,41.99222,41.99329,41.993173,41.99315,41.990054,41.990993,41.990523,41.993062,41.991574,41.992303,41.992159,41.991708,41.992522,41.99171,41.990134,41.9887374,41.987686,41.98651,41.990921],[-93.6193873,-93.61732,-93.638933,-93.638925,-93.633792,-93.633826,-93.632852,-93.639068,-93.636947,-93.638647,-93.634626,-93.632913,-93.62655,-93.6235954,-93.622971,-93.636655,-93.636372,-93.639366,-93.626537,-93.628806,-93.627112,-93.622769,-93.624373,-93.627271,-93.62728,-93.627232,-93.625924,-93.625966,-93.625848,-93.653201,-93.654645,-93.6537329,-93.652713,-93.652336,-93.655051,-93.656067,-93.654853,-93.654121,-93.649447,-93.651013,-93.644356,-93.641635,-93.642252,-93.651381,-93.652334,-93.654206,-93.653616,-93.650578,-93.652972,-93.642368,-93.650318,-93.647105,-93.642224,-93.692412,-93.68981,-93.687694,-93.686263,-93.686264,-93.68628,-93.685028,-93.685029,-93.677205,-93.673669,-93.672096,-93.672067,-93.660672,-93.651773,-93.652773,-93.647988,-93.64614,-93.645599,-93.644889,-93.644889,-93.64489,-93.6468879,-93.641253,-93.643556,-93.648143,-93.65921,-93.639516,-93.633351,-93.637463,-93.637788,-93.631435,-93.632554,-93.626137,-93.625926,-93.623383,-93.626939,-93.627536,-93.625728,-93.622904,-93.622864,-93.620651,-93.620504,-93.629287,-93.629313,-93.62475,-93.624601,-93.621431,-93.614244,-93.617553,-93.617117,-93.6177041,-93.616582,-93.614882,-93.614241,-93.613624,-93.615622,-93.617092,-93.61722,-93.618428,-93.615475,-93.611642,-93.6192014,-93.620341,-93.6116762,-93.607275,-93.606905,-93.61061,-93.605969,-93.609218,-93.607801,-93.605943,-93.606742,-93.604835,-93.603705,-93.617014,-93.62024,-93.620241,-93.6179112,-93.615431,-93.608954,-93.6090761,-93.610466,-93.610468,-93.606789,-93.606789,-93.610588,-93.61221,-93.607238,-93.608752,-93.607088,-93.625737,-93.625586,-93.622597,-93.623523,-93.622448,-93.620409,-93.62338,-93.621674,-93.621835,-93.621632,-93.629495,-93.626876,-93.620386,-93.656231,-93.658265,-93.656924,-93.658081,-93.657573,-93.672036,-93.671021,-93.671076,-93.673401,-93.672381,-93.678141,-93.676659,-93.677992,-93.676654,-93.676674,-93.673565,-93.672523,-93.670503,-93.670114,-93.663619,-93.667904,-93.675038,-93.671178,-93.662762,-93.66357,-93.667708,-93.66526,-93.6658876,-93.663367,-93.659808,-93.685988,-93.683167,-93.682026,-93.69019,-93.69245,-93.692521,-93.68329,-93.6801805,-93.682138,-93.69188,-93.688258,-93.68773,-93.687807,-93.687745,-93.683727,-93.687859,-93.685523,-93.687764,-93.680864,-93.682686,-93.681227,-93.6837,-93.687922,-93.685961,-93.6783313,-93.677482,-93.677545,-93.676372,-93.675268,-93.675175,-93.6663375,-93.66751,-93.664841,-93.655762,-93.655653,-93.6514413,-93.64678,-93.646683,-93.639781,-93.642793,-93.639678,-93.645014,-93.645729,-93.641677,-93.625291,-93.629496,-93.626689,-93.615272,-93.608271,-93.60775,-93.615993,-93.604947,-93.602823,-93.603972,-93.602002,-93.657894,-93.649203,-93.649096,-93.649471,-93.647645,-93.646814,-93.652495,-93.60743,-93.608372,-93.60359,-93.604318,-93.604425,-93.603435,-93.601806,-93.603398,-93.601449,-93.606856,-93.6035354,-93.600582,-93.599791,-93.600006,-93.600147,-93.619562,-93.639388,-93.6364,-93.637442,-93.633024,-93.636955,-93.637046,-93.637671,-93.635824,-93.635753,-93.636822,-93.636047,-93.634644,-93.633245,-93.628639,-93.629357,-93.636512,-93.637774,-93.63745,-93.638366,-93.634212,-93.637043,-93.637289,-93.635919,-93.639511,-93.6392308,-93.639366,-93.632915,-93.630253,-93.631432,-93.631791,-93.625856,-93.626554,-93.626603,-93.627037,-93.625657,-93.625542,-93.6234986,-93.6223289,-93.627396,-93.628119,-93.629456,-93.629851,-93.629881,-93.624729,-93.625945,-93.625918,-93.625846,-93.625688,-93.62569,-93.625696,-93.6256036,-93.626521,-93.658877,-93.6562749,-93.652716,-93.657828,-93.654241,-93.654144,-93.653955,-93.652863,-93.654439,-93.658224,-93.657835,-93.657049,-93.654437,-93.658854,-93.656737,-93.653152,-93.652884,-93.654724,-93.652084,-93.652724,-93.6541267,-93.6547754,-93.643771,-93.65604,-93.6545189,-93.65048,-93.649779,-93.649743,-93.650244,-93.655483,-93.654215,-93.654405,-93.6514849,-93.650238,-93.645752,-93.641561,-93.642094,-93.642911,-93.642914,-93.642749,-93.643312,-93.644106,-93.642626,-93.640051,-93.640065,-93.649649,-93.642813,-93.642932,-93.641919,-93.641377,-93.641313,-93.641303,-93.641397,-93.641312,-93.640879,-93.65429,-93.657163,-93.650957,-93.652608,-93.65238,-93.655555,-93.655988,-93.654048,-93.651503,-93.652117,-93.650529,-93.650822,-93.644121,-93.64247,-93.642553,-93.643944,-93.641251,-93.641296,-93.6504,-93.650417,-93.650377,-93.650338,-93.648275,-93.64287,-93.643842,-93.642474,-93.643752,-93.641501,-93.692017,-93.692472,-93.691506,-93.691707,-93.691068,-93.690159,-93.690402,-93.689136,-93.6888,-93.687843,-93.686267,-93.687804,-93.685037,-93.6790752,-93.680198,-93.676364,-93.677008,-93.675716,-93.669538,-93.673518,-93.669845,-93.660327,-93.646316,-93.646338,-93.647039,-93.646916,-93.644205,-93.645482,-93.641799,-93.645923,-93.65921,-93.635342,-93.634803,-93.631881,-93.631519,-93.633163,-93.632537,-93.632823,-93.637964,-93.637905,-93.632647,-93.631899,-93.628903,-93.627047,-93.624612,-93.630714,-93.627416,-93.622991,-93.623485,-93.622635,-93.623411,-93.628658,-93.6273127,-93.627836,-93.626979,-93.624581,-93.622685,-93.621699,-93.629499,-93.625633,-93.622644,-93.622494,-93.617963,-93.615686,-93.615602,-93.614619,-93.614216,-93.619029,-93.618784,-93.61488,-93.614768,-93.61588,-93.620016,-93.618478,-93.6165473,-93.618549,-93.6190376,-93.617219,-93.6124,-93.617207,-93.618605,-93.618607,-93.62008,-93.620336,-93.61719,-93.617182,-93.6146206,-93.6144209,-93.61239,-93.612241,-93.614049,-93.615465,-93.613899,-93.612253,-93.610355,-93.607655,-93.610588,-93.610649,-93.609226,-93.606573,-93.60552,-93.609059,-93.608977,-93.61061,-93.608903,-93.60891,-93.606891,-93.606889,-93.605968,-93.606786,-93.610605,-93.604987,-93.604047,-93.603879,-93.603701,-93.6179738,-93.619013,-93.6195025,-93.62026,-93.615598,-93.615599,-93.6195361,-93.618486,-93.6200683,-93.6176367,-93.615381,-93.608866,-93.607776,-93.608829,-93.606842,-93.606841,-93.610469,-93.606788,-93.6178679,-93.613887,-93.612245,-93.610582,-93.610044,-93.6076506,-93.608704,-93.607058,-93.605207,-93.628848,-93.6279815,-93.625968,-93.626116,-93.624564,-93.624567,-93.622513,-93.623515,-93.623645,-93.623644,-93.623629,-93.622565,-93.621351,-93.6213592,-93.62726,-93.621525,-93.621504,-93.621506,-93.621635,-93.621633,-93.628299,-93.626716,-93.621862,-93.625154,-93.599575,-93.659205,-93.658207,-93.658173,-93.657107,-93.6601886,-93.657241,-93.675401,-93.673382,-93.677991,-93.671849,-93.671369,-93.666432,-93.666813,-93.662126,-93.659279,-93.660877,-93.671177,-93.666242,-93.665996,-93.665994,-93.667776,-93.668212,-93.667209,-93.665271,-93.663366,-93.6641939,-93.661005,-93.685733,-93.685986,-93.68341,-93.681063,-93.682176,-93.679789,-93.679784,-93.6791403,-93.679109,-93.679784,-93.6795273,-93.6790969,-93.6791456,-93.6791251,-93.679181,-93.687067,-93.688859,-93.688993,-93.689091,-93.690266,-93.690971,-93.689072,-93.691433,-93.69157,-93.690111,-93.689066,-93.685125,-93.6825646,-93.691883,-93.692135,-93.688372,-93.690401,-93.691585,-93.6928668,-93.690548,-93.690324,-93.691667,-93.688383,-93.689431,-93.691175,-93.689438,-93.689942,-93.691672,-93.688628,-93.686916,-93.687212,-93.685068,-93.684954,-93.683157,-93.683159,-93.68317,-93.68351,-93.687819,-93.686218,-93.68269,-93.682623,-93.682983,-93.686744,-93.685673,-93.67915,-93.680706,-93.680782,-93.68218,-93.68167,-93.681297,-93.681439,-93.684115,-93.685333,-93.684536,-93.678305,-93.676372,-93.672234,-93.6739723,-93.676608,-93.669952,-93.666315,-93.664685,-93.6644556,-93.659191,-93.664812,-93.664336,-93.663302,-93.657963,-93.655674,-93.651825,-93.652027,-93.651786,-93.64857,-93.651669,-93.648414,-93.646655,-93.6459433,-93.648419,-93.644704,-93.644636,-93.6428055,-93.6397,-93.642371,-93.640592,-93.644295,-93.6439975,-93.642885,-93.640762,-93.641356,-93.645743,-93.645743,-93.645743,-93.645743,-93.644864,-93.643539,-93.626839,-93.625299,-93.616895,-93.6165709,-93.588227,-93.607252,-93.607055,-93.603717,-93.604716,-93.604491,-93.604519,-93.602424,-93.652119,-93.65315,-93.651832,-93.646455,-93.644884,-93.6446004,-93.646816,-93.651703,-93.6042638,-93.608359,-93.608196,-93.60821,-93.604252,-93.604218,-93.603479,-93.600636,-93.603111,-93.601975,-93.60176,-93.601127,-93.606688,-93.604928,-93.603867,-93.606925,-93.599866,-93.600191,-93.6188286,-93.6163289,-93.639062,-93.637796,-93.638775,-93.638778,-93.637482,-93.636718,-93.633876,-93.633781,-93.633797,-93.639007,-93.637763,-93.636798,-93.637279,-93.632013,-93.628831,-93.627911,-93.622299,-93.6225667,-93.6391494,-93.639715,-93.637825,-93.637808,-93.636511,-93.636505,-93.630299,-93.633168,-93.635153,-93.63719,-93.633905,-93.635216,-93.6341769,-93.630231,-93.626544,-93.627875,-93.630208,-93.624265,-93.628971,-93.627629,-93.627114,-93.629462,-93.62765,-93.627309,-93.629425,-93.627234,-93.625848,-93.62649,-93.658873,-93.6573126,-93.652941,-93.654642,-93.6566551,-93.657889,-93.657188,-93.656801,-93.654734,-93.654443,-93.656958,-93.65572,-93.652549,-93.651216,-93.655587,-93.654914,-93.654187,-93.6513669,-93.653868,-93.651845,-93.6514243,-93.649997,-93.650607,-93.65089,-93.640958,-93.6424399,-93.643039,-93.643964,-93.640051,-93.649587,-93.641331,-93.642397,-93.643632,-93.641342,-93.640959,-93.654594,-93.65412,-93.653324,-93.650932,-93.651562,-93.655279,-93.656036,-93.653929,-93.65579,-93.644107,-93.642334,-93.642092,-93.642376,-93.643976,-93.64403,-93.644024,-93.643788,-93.650397,-93.650358,-93.648255,-93.643836,-93.64119,-93.6430423,-93.643692,-93.641651,-93.644018,-93.639662,-93.692497,-93.692431,-93.691695,-93.691836,-93.690761,-93.690364,-93.691246,-93.689769,-93.689024,-93.689801,-93.688803,-93.687819,-93.686428,-93.686278,-93.674093,-93.675743,-93.673965,-93.672503,-93.670551,-93.66203,-93.660522,-93.653482,-93.653232,-93.647488,-93.646063,-93.646113,-93.64489,-93.644891,-93.644172,-93.644118,-93.646984,-93.646986,-93.646989,-93.647044,-93.643746,-93.643729,-93.643378,-93.641785,-93.64177,-93.641742,-93.647655,-93.648954,-93.655582,-93.635095,-93.634976,-93.631526,-93.631779,-93.631245,-93.6261431,-93.624762,-93.624632,-93.629079,-93.630152,-93.630732,-93.624499,-93.625799,-93.622673,-93.622662,-93.629555,-93.629884,-93.6277098,-93.6267969,-93.626171,-93.626762,-93.623267,-93.6219226,-93.621566,-93.62936,-93.627578,-93.628236,-93.6285065,-93.622644,-93.622644,-93.623556,-93.620487,-93.6167,-93.616552,-93.6178602,-93.6169916,-93.615685,-93.615614,-93.619022,-93.617544,-93.617543,-93.614856,-93.612394,-93.613383,-93.617238,-93.617084,-93.617086,-93.615621,-93.6165173,-93.618457,-93.615484,-93.61076,-93.612269,-93.612253,-93.607422,-93.610663,-93.605189,-93.605659,-93.608909,-93.60688,-93.605971,-93.60597,-93.606788,-93.607948,-93.608765,-93.608903,-93.6071073,-93.607799,-93.604986,-93.604837,-93.6195726,-93.619415,-93.615582,-93.62005,-93.616991,-93.6189373,-93.6195822,-93.6198478,-93.620219,-93.613969,-93.615447,-93.615031,-93.6112887,-93.607771,-93.606764,-93.607713,-93.607724,-93.606802,-93.6077128,-93.61844,-93.620183,-93.618297,-93.617386,-93.616885,-93.616998,-93.616996,-93.6096888,-93.610008,-93.608752,-93.6075521,-93.6059697,-93.606196,-93.604486,-93.628806,-93.6267116,-93.625728,-93.625926,-93.624715,-93.625584,-93.623666,-93.624566,-93.623514,-93.621532,-93.621532,-93.622102,-93.624688,-93.624687,-93.623632,-93.622552,-93.621485,-93.620392,-93.620391,-93.624591,-93.624471,-93.6218858,-93.627,-93.625556,-93.621676,-93.621065,-93.6259506,-93.628447,-93.6261621,-93.6265884,-93.6295,-93.625289,-93.623738,-93.624819,-93.6246096,-93.656979,-93.657958,-93.658395,-93.675551,-93.669806,-93.672241,-93.676873,-93.677717,-93.677945,-93.672718,-93.674179,-93.671905,-93.666954,-93.666912,-93.665521,-93.665355,-93.665356,-93.666979,-93.660812,-93.678125,-93.676219,-93.6739889,-93.67358,-93.666066,-93.666216,-93.666147,-93.66515,-93.663054,-93.660595,-93.6608625,-93.66526,-93.6644616,-93.6618779,-93.662924,-93.658528,-93.659949,-93.683223,-93.683302,-93.682029,-93.682032,-93.682175,-93.691284,-93.68698,-93.686789,-93.6826513,-93.679198,-93.692009,-93.691213,-93.691612,-93.690719,-93.690276,-93.690113,-93.688995,-93.689019,-93.691706,-93.688958,-93.68895,-93.685742,-93.686888,-93.685158,-93.685865,-93.6844,-93.684289,-93.683158,-93.687704,-93.684227,-93.684396,-93.682718,-93.685572,-93.686982,-93.687674,-93.683856,-93.67994,-93.68369,-93.679505,-93.68031,-93.686931,-93.685207,-93.684451,-93.684397,-93.683127,-93.682991,-93.681436,-93.681401,-93.678303,-93.678304,-93.677407,-93.673447,-93.673571,-93.673137,-93.671443,-93.672408,-93.675294,-93.673251,-93.670627,-93.6649923,-93.6653819,-93.665562,-93.661895,-93.660115,-93.664925,-93.665829,-93.663479,-93.66345,-93.661763,-93.6558,-93.656681,-93.655851,-93.64856,-93.651785,-93.651635,-93.647024,-93.646748,-93.648411,-93.646438,-93.646438,-93.6447083,-93.646288,-93.6447134,-93.646735,-93.642886,-93.641405,-93.63962,-93.639497,-93.641084,-93.643378,-93.64293,-93.640311,-93.640293,-93.645743,-93.645743,-93.628409,-93.627459,-93.625288,-93.626835,-93.625143,-93.625217,-93.615426,-93.616849,-93.604193,-93.616242,-93.616584,-93.615985,-93.615921,-93.607342,-93.607121,-93.60489,-93.604549,-93.602912,-93.639793,-93.650229,-93.653025,-93.651718,-93.649198,-93.649475,-93.650431,-93.644679,-93.646054,-93.6470597,-93.646726,-93.646645,-93.647834,-93.647214,-93.608267,-93.610145,-93.608197,-93.606374,-93.605351,-93.604649,-93.603601,-93.604269,-93.603178,-93.603458,-93.610109,-93.6051072,-93.604761,-93.603917,-93.603886,-93.608291,-93.600066,-93.577427,-93.618504,-93.618503,-93.617273,-93.61728,-93.63736,-93.636146,-93.636972,-93.634341,-93.636345,-93.637883,-93.639068,-93.637848,-93.637917,-93.635833,-93.633329,-93.634513,-93.631622,-93.629016,-93.629075,-93.622874,-93.629538,-93.628754,-93.628723,-93.637875,-93.638853,-93.637876,-93.637816,-93.634665,-93.633973,-93.631528,-93.632669,-93.638554,-93.634714,-93.634581,-93.636497,-93.633898,-93.633757,-93.632191,-93.626331,-93.629118,-93.626722,-93.627112,-93.6247461,-93.621646,-93.624798,-93.624798,-93.624431,-93.629747,-93.628656,-93.627855,-93.629777,-93.627242,-93.624575,-93.6257971,-93.6588809,-93.658015,-93.6566403,-93.654243,-93.6570383,-93.6566568,-93.6562702,-93.657967,-93.654774,-93.6570542,-93.6562861,-93.657444,-93.65331,-93.657211,-93.657235,-93.654689,-93.654617,-93.650917,-93.657492,-93.6565389,-93.65699,-93.6545189,-93.650597,-93.650499,-93.65045,-93.650406,-93.649838,-93.649808,-93.649798,-93.649397,-93.649319,-93.649291,-93.653414,-93.652565,-93.652054,-93.6514105,-93.650608,-93.646937,-93.645875,-93.64573,-93.644377,-93.644376,-93.644375,-93.644374,-93.644447,-93.644366,-93.641795,-93.641716,-93.640513,-93.642177,-93.644256,-93.644064,-93.643062,-93.643961,-93.643954,-93.639977,-93.64021,-93.64334,-93.641279,-93.642836,-93.643641,-93.6575919,-93.652894,-93.652641,-93.649901,-93.657271,-93.655159,-93.655905,-93.652946,-93.644118,-93.643769,-93.642069,-93.64176,-93.642261,-93.642243,-93.644086,-93.643836,-93.64248,-93.642344,-93.643544,-93.641251,-93.643748,-93.642275,-93.650341,-93.650299,-93.647584,-93.647933,-93.643974,-93.643792,-93.642046,-93.641233,-93.639574,-93.69179,-93.68912,-93.689767,-93.688804,-93.686428,-93.686413,-93.685185,-93.6791142,-93.677007,-93.673987,-93.673889,-93.669559,-93.669552,-93.672318,-93.673518,-93.671941,-93.661848,-93.661983,-93.653082,-93.652658,-93.647974,-93.64733,-93.647166,-93.646637,-93.646607,-93.646576,-93.646478,-93.644107,-93.644096,-93.645671,-93.641813,-93.639931,-93.639878,-93.643456,-93.648204,-93.645589,-93.64577,-93.648172,-93.645547,-93.645544,-93.658232,-93.6574973,-93.634545,-93.634528,-93.635899,-93.631554,-93.638051,-93.63687,-93.631673,-93.631302,-93.625898,-93.624745,-93.623104,-93.621344,-93.62232,-93.622834,-93.621452,-93.626945,-93.630023,-93.6279875,-93.6276375,-93.624484,-93.62326,-93.621368,-93.622804,-93.621503,-93.629545,-93.628654,-93.623704,-93.621485,-93.620497,-93.619383,-93.61706,-93.616099,-93.618143,-93.615535,-93.618786,-93.619352,-93.617521,-93.615281,-93.613046,-93.612347,-93.613457,-93.613983,-93.6192677,-93.617242,-93.615622,-93.618428,-93.617075,-93.612401,-93.612075,-93.618606,-93.6197518,-93.620343,-93.618445,-93.615616,-93.615485,-93.6133009,-93.615536,-93.612418,-93.6139,-93.613899,-93.607117,-93.609148,-93.610561,-93.605388,-93.610609,-93.610611,-93.607963,-93.606932,-93.605968,-93.60674,-93.606782,-93.605942,-93.606746,-93.604836,-93.603701,-93.603703,-93.604562,-93.6190545,-93.62024,-93.617135,-93.6176729,-93.615432,-93.615446,-93.60861,-93.608868,-93.606857,-93.606842,-93.61047,-93.606789,-93.618662,-93.6190848,-93.617061,-93.618294,-93.616911,-93.6188503,-93.617037,-93.612061,-93.612191,-93.608882,-93.6094532,-93.608678,-93.6047622,-93.6041765,-93.6051157,-93.6281873,-93.6262716,-93.627605,-93.627487,-93.624715,-93.623521,-93.621501,-93.623643,-93.623631,-93.624539,-93.621502,-93.6224,-93.622402,-93.6212119,-93.6209132,-93.62576,-93.622147,-93.6220032,-93.628447,-93.629496,-93.6277573,-93.629501,-93.6274918,-93.628263,-93.6263765,-93.622793,-93.62188,-93.623814,-93.655699,-93.656124,-93.657089,-93.655924,-93.675549,-93.675559,-93.669741,-93.674417,-93.67369,-93.677504,-93.676953,-93.675714,-93.677997,-93.677601,-93.676681,-93.668544,-93.664037,-93.662239,-93.660671,-93.660691,-93.661983,-93.661985,-93.676272,-93.669606,-93.671221,-93.671511,-93.66504,-93.663415,-93.66712,-93.66061,-93.6644275,-93.66468,-93.6643138,-93.658545,-93.66057,-93.659659,-93.658537,-93.693153,-93.684102,-93.684102,-93.683309,-93.683226,-93.681061,-93.686828,-93.686941,-93.690342,-93.69067,-93.691248,-93.691213,-93.689071,-93.689071,-93.69003,-93.688941,-93.688923,-93.689884,-93.688917,-93.684944,-93.682439,-93.679216,-93.683612,-93.691971,-93.692069,-93.690402,-93.690545,-93.692696,-93.692415,-93.690633,-93.688952,-93.689344,-93.689518,-93.691469,-93.692052,-93.684962,-93.686678,-93.687734,-93.684837,-93.684993,-93.683235,-93.684354,-93.683158,-93.683169,-93.687856,-93.686575,-93.682805,-93.686795,-93.687786,-93.68675,-93.683813,-93.6800822,-93.679805,-93.681274,-93.679505,-93.687684,-93.684137,-93.684714,-93.684625,-93.684285,-93.684271,-93.683732,-93.6783521,-93.677545,-93.676222,-93.671455,-93.671454,-93.671453,-93.673197,-93.6747048,-93.67525,-93.674327,-93.673253,-93.6762195,-93.676241,-93.665737,-93.664768,-93.663509,-93.664686,-93.661542,-93.660303,-93.665875,-93.66346,-93.663455,-93.663329,-93.646688,-93.644669,-93.6447083,-93.644511,-93.644698,-93.64476,-93.646289,-93.643215,-93.64281,-93.644426,-93.641322,-93.642359,-93.640497,-93.641156,-93.642663,-93.6440062,-93.640326,-93.645785,-93.643548,-93.643429,-93.640182,-93.628412,-93.6295,-93.597005,-93.626875,-93.62826,-93.625313,-93.629496,-93.628236,-93.626689,-93.613886,-93.615697,-93.615618,-93.606577,-93.606672,-93.607597,-93.607551,-93.606933,-93.60104,-93.647403,-93.650588,-93.650569,-93.650561,-93.651732,-93.650417,-93.644366,-93.646782,-93.646791,-93.646788,-93.646894,-93.646634,-93.647585,-93.647199,-93.662162,-93.653109,-93.652429,-93.608192,-93.6072813,-93.607395,-93.605355,-93.605354,-93.601507,-93.602094,-93.603422,-93.604923,-93.604644,-93.604521,-93.60678,-93.607114,-93.604858,-93.605119,-93.604146,-93.604858,-93.618182,-93.616439,-93.617432,-93.639067,-93.637566,-93.636378,-93.636052,-93.635967,-93.635935,-93.635932,-93.631604,-93.635716,-93.636949,-93.639293,-93.634231,-93.631572,-93.632571,-93.629337,-93.629174,-93.628607,-93.62877,-93.626337,-93.626477,-93.6220148,-93.625169,-93.63007,-93.628269,-93.638026,-93.637874,-93.6326645,-93.631515,-93.633002,-93.635486,-93.6353259,-93.636044,-93.635367,-93.630348,-93.632355,-93.630289,-93.631162,-93.62626,-93.625373,-93.627849,-93.624411,-93.627103,-93.627328,-93.629846,-93.629516,-93.628857,-93.624767,-93.658864,-93.658413,-93.658226,-93.6566537,-93.654774,-93.657984,-93.658788,-93.652334,-93.654831,-93.653452,-93.6531488,-93.643412,-93.643862,-93.657883,-93.653084,-93.650626,-93.6534099,-93.649714,-93.654138,-93.649525,-93.654124,-93.65414,-93.650617,-93.645774,-93.644411,-93.644306,-93.640287,-93.640227,-93.641321,-93.6419,-93.642261,-93.642241,-93.642116,-93.641518,-93.641522,-93.64458,-93.642588,-93.6436049,-93.640211,-93.642713,-93.642913,-93.641281,-93.64136,-93.641283,-93.641053,-93.657851,-93.6532,-93.650059,-93.653915,-93.655997,-93.655704,-93.653608,-93.653479,-93.642317,-93.64178,-93.641793,-93.643635,-93.643811,-93.650361,-93.64407,-93.643898,-93.642862,-93.642309,-93.642374,-93.64357,-93.643173,-93.643859,-93.643977,-93.692575,-93.691403,-93.691787,-93.691754,-93.690959,-93.691246,-93.689021,-93.687695,-93.686267,-93.68665,-93.681407,-93.675963,-93.677123,-93.675723,-93.677012,-93.671037,-93.672411,-93.670919,-93.669708,-93.669757,-93.671926,-93.660643,-93.653876,-93.653512,-93.650474,-93.650057,-93.647832,-93.646526,-93.644891,-93.644891,-93.644891,-93.644891,-93.645617,-93.643373,-93.648931,-93.649329,-93.649352,-93.649329,-93.658237,-93.633752,-93.6346,-93.634738,-93.633466,-93.630258,-93.633098,-93.627935,-93.625976,-93.626097,-93.626161,-93.627517,-93.628069,-93.6269,-93.626345,-93.624612,-93.629155,-93.623414,-93.623564,-93.622312,-93.630156,-93.626765,-93.627065,-93.6282949,-93.621621,-93.621362,-93.622017,-93.621399,-93.628568,-93.629256,-93.6270261,-93.624738,-93.616852,-93.615629,-93.6181153,-93.620355,-93.618389,-93.619334,-93.618625,-93.612886,-93.612566,-93.614251,-93.612462,-93.618319,-93.618593,-93.612464,-93.613491,-93.6136767,-93.615474,-93.61547,-93.612263,-93.620354,-93.617182,-93.6143498,-93.613965,-93.610772,-93.612239,-93.6116427,-93.612266,-93.614049,-93.612404,-93.613899,-93.608909,-93.608984,-93.607582,-93.60942,-93.610278,-93.610501,-93.607257,-93.606508,-93.605727,-93.609054,-93.608903,-93.608904,-93.606743,-93.609053,-93.609053,-93.609053,-93.606892,-93.606893,-93.606744,-93.603704,-93.605839,-93.604568,-93.6050674,-93.6195519,-93.617139,-93.618489,-93.619513,-93.615681,-93.618335,-93.615638,-93.614011,-93.614615,-93.613905,-93.607862,-93.608833,-93.610354,-93.608938,-93.607744,-93.606803,-93.613889,-93.615365,-93.6127958,-93.613763,-93.608901,-93.610438,-93.609665,-93.606195,-93.626022,-93.625576,-93.620433,-93.623642,-93.62454,-93.624553,-93.623481,-93.621484,-93.620411,-93.6243778,-93.622386,-93.625436,-93.621562,-93.622624,-93.626858,-93.626631,-93.6275675,-93.6273219,-93.628416,-93.6265888,-93.65565,-93.65815,-93.6578344,-93.6600396,-93.655885,-93.655786,-93.67555,-93.675399,-93.675408,-93.675606,-93.672278,-93.674385,-93.673601,-93.672645,-93.670204,-93.670514,-93.665569,-93.66563,-93.667081,-93.669544,-93.66412,-93.668392,-93.667082,-93.662039,-93.678165,-93.676352,-93.674402,-93.676681,-93.666035,-93.665153,-93.663475,-93.661562,-93.660664,-93.660227,-93.666095,-93.666096,-93.660187,-93.6627359,-93.658506,-93.685537,-93.683235,-93.68218,-93.689071,-93.690366,-93.691234,-93.688925,-93.688924,-93.689485,-93.689084,-93.688934,-93.68216,-93.681349,-93.682095,-93.683877,-93.680401,-93.691896,-93.688302,-93.690363,-93.692221,-93.690215,-93.68923,-93.691243,-93.688577,-93.686825,-93.687158,-93.683468,-93.683246,-93.681378,-93.687703,-93.686288,-93.685504,-93.685697,-93.685571,-93.683718,-93.679147,-93.681479,-93.679803,-93.682686,-93.683728,-93.683777,-93.687684,-93.687685,-93.687826,-93.684089,-93.684116,-93.684116,-93.685369,-93.683435,-93.684332,-93.683758,-93.678267,-93.676369,-93.676361,-93.675297,-93.675548,-93.672379,-93.671304,-93.671612,-93.670025,-93.669977,-93.669927,-93.670063,-93.670039,-93.669985,-93.669924,-93.661951,-93.659182,-93.659222,-93.660119,-93.663458,-93.661995,-93.656753,-93.659043,-93.655631,-93.648562,-93.651811,-93.651285,-93.646942,-93.646438,-93.648417,-93.648416,-93.64644,-93.6447118,-93.647027,-93.644307,-93.644258,-93.64048,-93.643443,-93.641155,-93.642319,-93.641002,-93.6416718,-93.640612,-93.642884,-93.644203,-93.639428,-93.643489,-93.628467,-93.62841,-93.628298,-93.629496,-93.626683,-93.625147,-93.615307,-93.61385,-93.615012,-93.604344,-93.616026,-93.616242,-93.599969,-93.606736,-93.603671,-93.603173,-93.601191,-93.603854,-93.644527,-93.649773,-93.646099,-93.646575,-93.646172,-93.646758,-93.64789,-93.610268,-93.608343,-93.608195,-93.608195,-93.607142,-93.604391,-93.603922,-93.604279,-93.601089,-93.603181,-93.603524,-93.600439,-93.6019,-93.601615,-93.603534,-93.6086884,-93.606842,-93.606847,-93.60019],3,null,null,{"interactive":true,"className":"","stroke":true,"color":"white","weight":5,"opacity":0.4,"fill":true,"fillColor":"white","fillOpacity":0.2},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null,null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641487],"lat":[42.060936,42.060977]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641488],"lat":[42.060936,42.061114]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641772],"lat":[42.060936,42.061149]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.641781],"lat":[42.060936,42.061318]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]},{"method":"addPolylines","args":[[[[{"lng":[-93.641325,-93.642447],"lat":[42.060936,42.061193]}]]],null,null,{"interactive":true,"className":"","stroke":true,"color":"yellow","weight":1,"opacity":0.5,"fill":false,"fillColor":"yellow","fillOpacity":0.2,"smoothFactor":1,"noClip":false},null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]}],"limits":{"lat":[41.986509,42.063388],"lng":[-93.693153,-93.577427]}},"evals":[],"jsHooks":[]}</script>

---

# Resampling a 5-NN model


```r
five_nn &lt;- 
  nearest_neighbor(neighbors = 5) %&gt;% 
  set_engine("kknn") %&gt;% 
  set_mode("regression")
```

If we were using this model with the training set: 


```r
five_nn %&gt;% fit(log10(Sale_Price) ~ Longitude + Latitude, data = ames_train)
```

```
## parsnip model object
## 
## Fit in:  31ms
## Call:
## kknn::train.kknn(formula = formula, data = data, ks = ~5)
## 
## Type of response variable: continuous
## minimal mean absolute error: 0.06753097
## Minimal mean squared error: 0.009633708
## Best kernel: optimal
## Best k: 5
```

---

# Resampling a 5-NN model

Let's repeat that but using each of the 10 analysis sets:


```r
cv_splits_knn &lt;-
  cv_splits %&gt;%
  mutate(
    fits = 
      map(
        splits,
        ~ five_nn %&gt;% fit(log10(Sale_Price) ~ Longitude + Latitude, data = analysis(.x))    
      )
  ) 
cv_splits_knn %&gt;% slice(1:3)
```

```
## #  10-fold cross-validation 
## # A tibble: 3 x 3
##   splits           id     fits    
## * &lt;list&gt;           &lt;chr&gt;  &lt;list&gt;  
## 1 &lt;split [2K/220]&gt; Fold01 &lt;fit[+]&gt;
## 2 &lt;split [2K/220]&gt; Fold02 &lt;fit[+]&gt;
## 3 &lt;split [2K/220]&gt; Fold03 &lt;fit[+]&gt;
```

---

# Predictions for each model

.pull-left[

.font90[


```r
*knn_pred &lt;-
* map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,
*          ~ predict(.x, assessment(.y)),
*          .id = "fold")
```

]

]
.pull-right[

```
## # A tibble: 2,199 x 2
##    fold  .pred
##    &lt;chr&gt; &lt;dbl&gt;
##  1 1      5.36
##  2 1      5.28
##  3 1      5.27
##  4 1      5.06
##  5 1      5.37
##  6 1      5.30
##  7 1      5.14
##  8 1      5.08
##  9 1      5.10
## 10 1      5.24
## # … with 2,189 more rows
```
]


---

# Predictions for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

*prices &lt;-
* map_dfr(cv_splits_knn$splits,
*         ~ assessment(.x) %&gt;% select(Sale_Price))
```

]

]
.pull-right[

```
## # A tibble: 2,199 x 1
##    Sale_Price
##         &lt;int&gt;
##  1     216000
##  2     160000
##  3     216500
##  4     133000
##  5     254000
##  6     218500
##  7     108000
##  8     105000
##  9     132000
## 10     155000
## # … with 2,189 more rows
```
]


---

# Predictions for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
* mutate(Sale_Price = log10(Sale_Price))
```

]

]
.pull-right[

```
## # A tibble: 2,199 x 1
##    Sale_Price
##         &lt;dbl&gt;
##  1       5.33
##  2       5.20
##  3       5.34
##  4       5.12
##  5       5.40
##  6       5.34
##  7       5.03
##  8       5.02
##  9       5.12
## 10       5.19
## # … with 2,189 more rows
```
]


---

# Compute RMSE for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
  mutate(Sale_Price = log10(Sale_Price))

*rmse_estimates &lt;-
* knn_pred %&gt;%
* bind_cols(prices)
```

]

]
.pull-right[

```
## # A tibble: 2,199 x 3
##    fold  .pred Sale_Price
##    &lt;chr&gt; &lt;dbl&gt;      &lt;dbl&gt;
##  1 1      5.36       5.33
##  2 1      5.28       5.20
##  3 1      5.27       5.34
##  4 1      5.06       5.12
##  5 1      5.37       5.40
##  6 1      5.30       5.34
##  7 1      5.14       5.03
##  8 1      5.08       5.02
##  9 1      5.10       5.12
## 10 1      5.24       5.19
## # … with 2,189 more rows
```
]


---

# Compute RMSE for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates &lt;- 
  knn_pred %&gt;%  
  bind_cols(prices) %&gt;% 
* group_by(fold)
```

]

]
.pull-right[

```
## # A tibble: 2,199 x 3
## # Groups:   fold [10]
##    fold  .pred Sale_Price
##    &lt;chr&gt; &lt;dbl&gt;      &lt;dbl&gt;
##  1 1      5.36       5.33
##  2 1      5.28       5.20
##  3 1      5.27       5.34
##  4 1      5.06       5.12
##  5 1      5.37       5.40
##  6 1      5.30       5.34
##  7 1      5.14       5.03
##  8 1      5.08       5.02
##  9 1      5.10       5.12
## 10 1      5.24       5.19
## # … with 2,189 more rows
```
]

---

# Compute RMSE for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates &lt;- 
  knn_pred %&gt;%  
  bind_cols(prices) %&gt;% 
  group_by(fold) %&gt;% 
* do(rmse = rmse(., Sale_Price, .pred))
```

]

]
.pull-right[

```
## Source: local data frame [10 x 2]
## Groups: &lt;by row&gt;
## 
## # A tibble: 10 x 2
##    fold  rmse            
##  * &lt;chr&gt; &lt;list&gt;          
##  1 1     &lt;tibble [1 × 3]&gt;
##  2 10    &lt;tibble [1 × 3]&gt;
##  3 2     &lt;tibble [1 × 3]&gt;
##  4 3     &lt;tibble [1 × 3]&gt;
##  5 4     &lt;tibble [1 × 3]&gt;
##  6 5     &lt;tibble [1 × 3]&gt;
##  7 6     &lt;tibble [1 × 3]&gt;
##  8 7     &lt;tibble [1 × 3]&gt;
##  9 8     &lt;tibble [1 × 3]&gt;
## 10 9     &lt;tibble [1 × 3]&gt;
```

```
## # A tibble: 1 x 3
##   .metric .estimator .estimate
##   &lt;chr&gt;   &lt;chr&gt;          &lt;dbl&gt;
## 1 rmse    standard       0.113
```
]

---

# Compute RMSE for each model

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates &lt;- 
  knn_pred %&gt;%  
  bind_cols(prices) %&gt;% 
  group_by(fold) %&gt;% 
  do(rmse = rmse(., Sale_Price, .pred)) %&gt;% 
* unnest(cols = c(rmse))
```

]

]
.pull-right[

```
## # A tibble: 10 x 4
##    fold  .metric .estimator .estimate
##    &lt;chr&gt; &lt;chr&gt;   &lt;chr&gt;          &lt;dbl&gt;
##  1 1     rmse    standard      0.113 
##  2 10    rmse    standard      0.0874
##  3 2     rmse    standard      0.0895
##  4 3     rmse    standard      0.102 
##  5 4     rmse    standard      0.0888
##  6 5     rmse    standard      0.0911
##  7 6     rmse    standard      0.101 
##  8 7     rmse    standard      0.103 
##  9 8     rmse    standard      0.0971
## 10 9     rmse    standard      0.113
```
]


---

# Compute Overall RMSE estimate

.pull-left[

.font90[


```r
knn_pred &lt;-                                              
  map2_dfr(cv_splits_knn$fits, cv_splits_knn$splits,     
           ~ predict(.x, assessment(.y)),                
           .id = "fold")                                 

prices &lt;-  
  map_dfr(cv_splits_knn$splits,  
          ~ assessment(.x) %&gt;% select(Sale_Price)) %&gt;%  
  mutate(Sale_Price = log10(Sale_Price))

rmse_estimates &lt;- 
  knn_pred %&gt;%  
  bind_cols(prices) %&gt;% 
  group_by(fold) %&gt;% 
  do(rmse = rmse(., Sale_Price, .pred)) %&gt;% 
  unnest(cols = c(rmse)) 

*mean(rmse_estimates$.estimate)
```

]

]
.pull-right[

```
## [1] 0.09851353
```
]

---

# How you probably feel right now

&lt;img src="images/exhausted.jpg" width="90%" /&gt;

---

# Resampling Notes

This is actually a _simple case_:

 * There is no attached recipe to be prepared for each resample
 
 * It assumes that 5 neighbors is optimal. 
 
Common Questions: 

 * Q: What happens to these 10 models? Is this some ensemble thing? 
 
   * A: They are only used for measuring performance so `/dev/null`
 
 * Q: What were we measuring again?  
 
   * A: The resampling estimate is how we think a 5-NN model fit on the _entire data set_ would preform.
 
 * Q: Do you actually expect us to do anything like this again? 
 
   * A: No. 
 

---

# Please don't be mad about the next few slides

&lt;img src="images/sorry.jpg" width="40%" style="display: block; margin: auto;" /&gt;


---

# Easy resampling using the {tune} package

There is a `fit_resamples()` function in the `tune` package that does all of this for you. 

.pull-left[

```r
library(tune)
easy_eval &lt;- 
  fit_resamples(
    log10(Sale_Price) ~ Longitude + Latitude,
    five_nn,
    resamples = cv_splits,
    control = control_resamples(save_pred = TRUE)
  )
```

This does the same process that we did manually except the resamples models are not saved (but you could save them). 

]
.pull-right[
.font80[


```r
easy_eval
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 5
##    splits           id     .metrics         .notes           .predictions      
##  * &lt;list&gt;           &lt;chr&gt;  &lt;list&gt;           &lt;list&gt;           &lt;list&gt;            
##  1 &lt;split [2K/220]&gt; Fold01 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  2 &lt;split [2K/220]&gt; Fold02 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  3 &lt;split [2K/220]&gt; Fold03 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  4 &lt;split [2K/220]&gt; Fold04 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  5 &lt;split [2K/220]&gt; Fold05 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  6 &lt;split [2K/220]&gt; Fold06 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  7 &lt;split [2K/220]&gt; Fold07 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  8 &lt;split [2K/220]&gt; Fold08 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
##  9 &lt;split [2K/220]&gt; Fold09 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [220 × 3]&gt;
## 10 &lt;split [2K/219]&gt; Fold10 &lt;tibble [2 × 3]&gt; &lt;tibble [0 × 1]&gt; &lt;tibble [219 × 3]&gt;
```

]

]


---

# Getting the statistics and predictions

.pull-left[

```r
collect_predictions(easy_eval) %&gt;% 
  arrange(.row) %&gt;% 
  slice(1:5) 
```

```
## # A tibble: 5 x 4
##   id     .pred  .row Sale_Price
##   &lt;chr&gt;  &lt;dbl&gt; &lt;int&gt;      &lt;dbl&gt;
## 1 Fold10  5.22     1       5.24
## 2 Fold10  5.34     2       5.39
## 3 Fold10  5.28     3       5.28
## 4 Fold09  5.28     4       5.29
## 5 Fold04  5.43     5       5.33
```

```r
collect_metrics(easy_eval)
```

```
## # A tibble: 2 x 5
##   .metric .estimator   mean     n std_err
##   &lt;chr&gt;   &lt;chr&gt;       &lt;dbl&gt; &lt;int&gt;   &lt;dbl&gt;
## 1 rmse    standard   0.0985    10 0.00298
## 2 rsq     standard   0.698     10 0.0153
```

]
.pull-right[

```r
collect_metrics(easy_eval, summarize = FALSE) %&gt;% 
  slice(1:10)
```

```
## # A tibble: 10 x 4
##    id     .metric .estimator .estimate
##    &lt;chr&gt;  &lt;chr&gt;   &lt;chr&gt;          &lt;dbl&gt;
##  1 Fold01 rmse    standard      0.113 
##  2 Fold01 rsq     standard      0.640 
##  3 Fold02 rmse    standard      0.0895
##  4 Fold02 rsq     standard      0.724 
##  5 Fold03 rmse    standard      0.102 
##  6 Fold03 rsq     standard      0.713 
##  7 Fold04 rmse    standard      0.0888
##  8 Fold04 rsq     standard      0.767 
##  9 Fold05 rmse    standard      0.0911
## 10 Fold05 rsq     standard      0.717
```

Note that the outcome data are already logged.

]


---
layout: false
class: inverse, middle, center

# Model Tuning


---

# Tuning Parameters

There are some models with parameters that _cannot be directly estimated from the data_. 

For example:

* The number of neighbors in a K-NN models.

* The depth of a classification tree. 

* The link function in a generalized linear model (e.g. logit, probit, etc).

* The covariance structure in a linear mixed model. 

---

# Tuning Parameters and Overfitting

Overfitting occurs when a model inappropriately picks up on trends in the training set that do not generalize to new samples. 

When this occurs, assessments of the model based on the training set can show good performance that does not reproduce in future samples.   

For example, _K_ = 1 neighbors is much more likely to overfit the data than larger values since they average more values. 

Also, how would you evaluate this model by re-predicting the training set? Those values would be optimistic since one of your neighbors is always you. 


---

# Model Tuning

.pull-left[

Unsurprisingly, we will evaluate a tuning parameter by fitting a model on one set of data and assessing it with another. 

_Grid search_ uses a pre-defined set of candidate tuning parameter values and evaluates their performance so that the best values can be used in the final model. 

We'll use resampling to do this. If there are _B_ resamples and _C_ tuning parameter combinations, we end up fitting  `\(B \times C\)`  models (but these can be done in parallel). 

]
.pull-right[

&lt;img src="images/TrainAlgo.png" width="120%" style="display: block; margin: auto;" /&gt;


]


---

# Better API's using the tune package

The `tune` package has more general functions for tuning models. There are two main strategies used: 

 * Grid search (as shown above) where all of the candidate models are known at the start. We pick the best of these. 

 * Iterative search where each iterations finds novel tuning parameter values to evaluate. 

Both have their advantages and disadvantages. At first, we will focus on grid search. 


---

# Elements Required for Grid Search &lt;img src="images/dials.png" class="title-hex"&gt;

.pull-left[
* A set of candidate values to evaluate.

* Measure of model performance.  

* A resampling scheme to reliably estimate model performance

We've discussed the last two. Now let's focus on the grid. 

There are two main types of grids: regular and non-regular. 

]
.pull-right[
&lt;img src="images/part-4-grid-plot-1.svg" width="100%" style="display: block; margin: auto;" /&gt;
]

 
---

# About Regular Grids

Usually combinatorial representation of vectors of tuning parameter values. Note that:

* The number of values don't have to be the same per parameter.

* The values can be regular on a transformed scale (e.g. log-10 for penalty).

* Quantitative and qualitative parameters can be combined.

* As the number of parameters increase, the curse of dimensionality kicks in.

* _Thought_ to be really inefficient but not in all cases (see the sub-model trick and `multi_predict()`).

* Bad when performance plateaus over a range of one or more parameters.


---

# Making Regular Grids &lt;img src="images/dials.png" class="title-hex"&gt;

.pull-left[
You can use `expand.grid()` to make a data frame where parameters are in columns and candidate values are in rows. 

`dials` has functions to create parameters and parameter sets. For example:


```r
penalty()
```

```
## Amount of Regularization  (quantitative)
## Transformer:  log-10 
## Range (transformed scale): [-10, 0]
```

```r
mixture()
```

```
## Proportion of lasso Penalty  (quantitative)
## Range: [0, 1]
```

```r
glmn_set &lt;- parameters(penalty(), mixture())
glmn_set
```

```
## Collection of 2 parameters for tuning
## 
##       id parameter type object class
##  penalty        penalty    nparam[+]
##  mixture        mixture    nparam[+]
```
]
.pull-right[

```r
glmn_grid &lt;- 
  grid_regular(glmn_param, levels = c(10, 5))
glmn_grid %&gt;% slice(1:4)
```

```
## # A tibble: 4 x 2
##         penalty mixture
## *         &lt;dbl&gt;   &lt;dbl&gt;
## 1 0.0000000001        0
## 2 0.00000000129       0
## 3 0.0000000167        0
## 4 0.000000215         0
```

Note that the grid for `penalty` is created in the log-10 space but the values in the data frame are in the original units. 
]


---

# Non-Regular Grids &lt;img src="images/dials.png" class="title-hex"&gt;

.pull-left[
There are two main methods that we have to make these: 

 * Random grids uniformly sample the parameter space (that might already be on a different scale).

 * Space-filling designs (SFD) are based on statistical experimental design principles and try to keep candidate values away from one another while encompassing the entire parameter space. 

There's no real downside to using the space-filling designs, so we will focus on these. 

]
.pull-right[
The code is easy when using a parameter set:


```r
set.seed(7454)
glmn_sfd &lt;- grid_max_entropy(glmn_param, size = 50)
glmn_sfd %&gt;% slice(1:4)
```

```
## # A tibble: 4 x 2
##      penalty mixture
##        &lt;dbl&gt;   &lt;dbl&gt;
## 1 0.00000362   0.172
## 2 0.0102       0.414
## 3 0.00000372   0.346
## 4 0.000709     0.544
```

```r
# grid_latin_hypercube() can also be used

# grid_random() too
```
]


---

# Modifying Parameter Sets &lt;img src="images/dials.png" class="title-hex"&gt;

.code70[

.pull-left[

```r
# The names can be changed:
glmn_set &lt;- parameters(lambda = penalty(), mixture())

# The ranges can also be set by their name:
glmn_set &lt;- 
  update(glmn_set, lambda = penalty(c(-5, -1)))
```

&lt;br&gt;


```r
# Some parameters depend on data dimensions:
mtry()
```

```
## # Randomly Selected Predictors  (quantitative)
## Range: [1, ?]
```

```r
rf_set &lt;- parameters(mtry(), trees())
```
]
.pull-right[

```r
rf_set
```

```
## Collection of 2 parameters for tuning
## 
##     id parameter type object class
##   mtry           mtry    nparam[?]
##  trees          trees    nparam[+]
## 
## Parameters needing finalization:
##    # Randomly Selected Predictors ('mtry')
## 
## See `?dials::finalize` or `?dials::update.parameters` for more information.
```

```r
# Sets the range of mtry to be the number of predictors
finalize(rf_set, mtcars %&gt;% select(-mpg))
```

```
## Collection of 2 parameters for tuning
## 
##     id parameter type object class
##   mtry           mtry    nparam[+]
##  trees          trees    nparam[+]
```
]

]


---

# Hands-On: K-NN Grids


Looking at the help file `?nearest_neighbors` and find the names of the three tuning parameters. 

Create a parameter set for these three, make at least one grid, and plot them. 

<div class="countdown" id="timer_5dcefba4" style="bottom:0;left:1;" data-warnwhen="0">
<code class="countdown-time"><span class="countdown-digits minutes">10</span><span class="countdown-digits colon">:</span><span class="countdown-digits seconds">00</span></code>
</div>


---

# K-NN Tuning parameters &lt;img src="images/dials.png" class="title-hex"&gt;

The `dist_power` parameter is related to the type of distance calculation. `dist_power = 2` is everyday Euclidean distance, `dist_power = 1` is Manhattan distance, and so on. Fractional values are acceptable. 

`weight_func` determines how much influence neighbors have based on their distance: 

&lt;img src="images/part-4-schemes-1.svg" width="100%" style="display: block; margin: auto;" /&gt;
 
 
---

# Tagging Tuning parameters &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;

To tune the model, the first step is to _tag_ the parameters that will be optimized. The `tune` package has a function called `tune()` that can be used to do this:


```r
library(tune)
knn_mod &lt;- 
  nearest_neighbor(neighbors = tune(), weight_func = tune()) %&gt;% 
  set_engine("kknn") %&gt;% 
  set_mode("regression")
```

`parameters` can detect these arguments:


```r
parameters(knn_mod)
```

```
## Collection of 2 parameters for tuning
## 
##           id parameter type object class
##    neighbors      neighbors    nparam[+]
##  weight_func    weight_func    dparam[+]
```


---

# Tagging Tuning parameters &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;

The `tune()` function has an optimal argument called `id` that can be used to name the parameters:


```r
nearest_neighbor(neighbors = tune("K"), weight_func = tune("weights")) %&gt;% 
  set_engine("kknn") %&gt;% 
  set_mode("regression") %&gt;% 
  parameters()
```

```
## Collection of 2 parameters for tuning
## 
##       id parameter type object class
##        K      neighbors    nparam[+]
##  weights    weight_func    dparam[+]
```

This mainly comes in handy when the same parameter type shows up in two different places (an example is shown later). 

Recipe objects can also have `tune()` in their arguments. 


---

# Grid Search &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;

Let's tune the model over a regular grid and optimize `neighbors` and `weight_func`. First we make the grid, then use the `tune_grid()` function:


```r
set.seed(522)
knn_reg &lt;- knn_mod %&gt;% parameters() %&gt;% grid_regular(levels = c(15, 5))
ctrl &lt;- control_grid(verbose = TRUE)

knn_reg_search &lt;- 
  tune_grid(ames_rec, model = knn_mod, resamples = cv_splits, grid = knn_reg, control = ctrl)
```

We are optimizing 75 models over 10 resamples. The output looks like:

.code70[

```
✓ Fold04: recipe
❯ Fold04: model 1/5
✓ Fold04: model 1/5
! Fold04: model 1/5 (predictions): some 'x' values beyond boundary knots may cause ill-conditioned bases
❯ Fold04: model 2/5
✓ Fold04: model 2/5
! Fold04: model 2/5 (predictions): some 'x' values beyond boundary knots may cause ill-conditioned bases
```

]

---

# Some Notes

 * If we are evaluating 75 models, why does it say `model 1/5`?
 
 * The recipe is made _once_ per resample to avoid redundant computations. 
 
 * The warnings about "ill-conditioned bases" is related to using splines for longitude and latitude. Remember those outlying houses on Lincoln Highway? 
 
   * These are printed _as they happen_ and are labeled so that you know which sub-model had the issue. 

 * Unfortunately, these messages don't show up when parallel processing is used (more on that later). We are working on this but it is a tough problem that I've been trying to solve for about a decade. 


---

# The Results  &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;

A tibble is returned that looks like the `rsample` object with an extra list column called `.metrics`:

.code70[

.pull-left[

```r
knn_reg_search
```

```
## #  10-fold cross-validation 
## # A tibble: 10 x 4
##    splits           id     .metrics           .notes          
##  * &lt;list&gt;           &lt;chr&gt;  &lt;list&gt;             &lt;list&gt;          
##  1 &lt;split [2K/220]&gt; Fold01 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
##  2 &lt;split [2K/220]&gt; Fold02 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
##  3 &lt;split [2K/220]&gt; Fold03 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
##  4 &lt;split [2K/220]&gt; Fold04 &lt;tibble [150 × 5]&gt; &lt;tibble [5 × 1]&gt;
##  5 &lt;split [2K/220]&gt; Fold05 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
##  6 &lt;split [2K/220]&gt; Fold06 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
##  7 &lt;split [2K/220]&gt; Fold07 &lt;tibble [150 × 5]&gt; &lt;tibble [5 × 1]&gt;
##  8 &lt;split [2K/220]&gt; Fold08 &lt;tibble [150 × 5]&gt; &lt;tibble [5 × 1]&gt;
##  9 &lt;split [2K/220]&gt; Fold09 &lt;tibble [150 × 5]&gt; &lt;tibble [5 × 1]&gt;
## 10 &lt;split [2K/219]&gt; Fold10 &lt;tibble [150 × 5]&gt; &lt;tibble [0 × 1]&gt;
```
]
.pull-right[

```r
# results for the first fold:
knn_reg_search$.metrics[[1]]
```

```
## # A tibble: 150 x 5
##    neighbors weight_func .metric .estimator .estimate
##        &lt;int&gt; &lt;chr&gt;       &lt;chr&gt;   &lt;chr&gt;          &lt;dbl&gt;
##  1         1 biweight    rmse    standard      0.111 
##  2         2 biweight    rmse    standard      0.103 
##  3         3 biweight    rmse    standard      0.0982
##  4         4 biweight    rmse    standard      0.0962
##  5         5 biweight    rmse    standard      0.0950
##  6         6 biweight    rmse    standard      0.0940
##  7         7 biweight    rmse    standard      0.0925
##  8         8 biweight    rmse    standard      0.0923
##  9         9 biweight    rmse    standard      0.0926
## 10        10 biweight    rmse    standard      0.0929
## # … with 140 more rows
```

150 rows = 75 models x 2 metrics
]

]


---

# Resampled Performance Estimates  &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/dplyr.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;

To get the overall resampling estimate (averaged over folds) for each parameter combination:

.code70[


```r
knn_perf &lt;- collect_metrics(knn_reg_search)
knn_perf %&gt;% slice(1:10)
```

```
## # A tibble: 10 x 7
##    neighbors weight_func  .metric .estimator   mean     n std_err
##        &lt;int&gt; &lt;chr&gt;        &lt;chr&gt;   &lt;chr&gt;       &lt;dbl&gt; &lt;int&gt;   &lt;dbl&gt;
##  1         1 biweight     rmse    standard   0.199     10 0.0512 
##  2         1 biweight     rsq     standard   0.508     10 0.0920 
##  3         1 epanechnikov rmse    standard   0.0971    10 0.00315
##  4         1 epanechnikov rsq     standard   0.717     10 0.0169 
##  5         1 rectangular  rmse    standard   0.0971    10 0.00315
##  6         1 rectangular  rsq     standard   0.717     10 0.0169 
##  7         1 triangular   rmse    standard   0.0971    10 0.00315
##  8         1 triangular   rsq     standard   0.717     10 0.0169 
##  9         1 triweight    rmse    standard   0.599     10 0.0557 
## 10         1 triweight    rsq     standard   0.0516    10 0.0111
```

]


---

# Resampled Performance Estimates  &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/ggplot2.png" class="title-hex"&gt;&lt;img src="images/dplyr.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;


```r
knn_perf %&gt;% 
  dplyr::filter(.metric == "rmse") %&gt;% 
  ggplot(aes(x = neighbors, y = mean, col = weight_func)) + 
  geom_point() + 
  geom_line()
```

&lt;img src="images/part-4-knn-plot-1.svg" width="50%" style="display: block; margin: auto;" /&gt;


---

# Zoomed  &lt;img src="images/parsnip.png" class="title-hex"&gt;&lt;img src="images/ggplot2.png" class="title-hex"&gt;&lt;img src="images/dplyr.png" class="title-hex"&gt;&lt;img src="images/dials.png" class="title-hex"&gt;


```r
knn_perf %&gt;% 
  dplyr::filter(.metric == "rmse") %&gt;% 
  ggplot(aes(x = neighbors, y = mean, col = weight_func)) + 
  geom_point() + 
  geom_line() + 
  ylim(c(0.08, 0.10))
```

&lt;img src="images/part-4-knn-plot-zoon-1.svg" width="50%" style="display: block; margin: auto;" /&gt;

---

# Finalizing the objects

If we felt that this MARS model was the best one, the recipe and model would be updated with the final parameter values and train them _on the entire training set_. Choosing the numerically best model:
 

```r
best_knn &lt;- select_best(knn_reg_search, metric = "rmse", maximize = FALSE)
best_knn
```

```
## # A tibble: 1 x 2
##   neighbors weight_func
##       &lt;int&gt; &lt;chr&gt;      
## 1         8 triangular
```

```r
final_ames_rec &lt;- 
  ames_rec %&gt;% 
  prep()

final_mars_mod &lt;- 
  knn_mod %&gt;%
  finalize_model(best_knn) %&gt;% 
  # Recall that sale price has been transformed by the recipe. 
  fit(Sale_Price ~ ., data = juice(final_ames_rec))
```
    </textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"ratio": "16:9",
"highlightLanguage": "R",
"highlightStyle": "github",
"highlightLines": true,
"countIncrementalSlides": false
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
  window.dispatchEvent(new Event('resize'));
});
(function(d) {
  var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
  if (!r) return;
  s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
  d.head.appendChild(s);
})(document);

(function(d) {
  var el = d.getElementsByClassName("remark-slides-area");
  if (!el) return;
  var slide, slides = slideshow.getSlides(), els = el[0].children;
  for (var i = 1; i < slides.length; i++) {
    slide = slides[i];
    if (slide.properties.continued === "true" || slide.properties.count === "false") {
      els[i - 1].className += ' has-continuation';
    }
  }
  var s = d.createElement("style");
  s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
  d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
  var deleted = false;
  slideshow.on('beforeShowSlide', function(slide) {
    if (deleted) return;
    var sheets = document.styleSheets, node;
    for (var i = 0; i < sheets.length; i++) {
      node = sheets[i].ownerNode;
      if (node.dataset["target"] !== "print-only") continue;
      node.parentNode.removeChild(node);
    }
    deleted = true;
  });
})();</script>

<script>
(function() {
  var links = document.getElementsByTagName('a');
  for (var i = 0; i < links.length; i++) {
    if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
      links[i].target = '_blank';
    }
  }
})();
</script>

<script>
slideshow._releaseMath = function(el) {
  var i, text, code, codes = el.getElementsByTagName('code');
  for (i = 0; i < codes.length;) {
    code = codes[i];
    if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
      text = code.textContent;
      if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
          /^\$\$(.|\s)+\$\$$/.test(text) ||
          /^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
        code.outerHTML = code.innerHTML;  // remove <code></code>
        continue;
      }
    }
    i++;
  }
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
  var script = document.createElement('script');
  script.type = 'text/javascript';
  script.src  = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
  if (location.protocol !== 'file:' && /^https?:/.test(script.src))
    script.src  = script.src.replace(/^https?:/, '');
  document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
  </body>
</html>