2_4_HyperparameterOptimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i.e. how to improve our model through **changing how our model is built**.\n",
    "\n",
    "<img src=\"fig/frankestein.jpg\" width=\"150\"> <img src=\"fig/terminator.jpg\" width=\"150\"> <img src=\"fig/wolverine.png\" width=\"150\">\n",
    "\n",
    " **Hello There!**         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four fairly common strategies to optimize one model's hyperparameters.\n",
    "\n",
    "<img src=\"fig/hyperparameter_meme_2.jpg\" width=\"400\">\n",
    "\n",
    "*List in descending order of coolness*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try applying some of these techniques to our titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "source('src/lib.R')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import training and testing dataset\n",
    "\n",
    "titanic_df = get_titanic_df()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grid Search\n",
    "\n",
    "This is the stupidest plossibly conveivable method.\n",
    "1. Try lots of combinations\n",
    "2. Select the best one\n",
    "\n",
    "It is also the simplest to implement and it is not a coincidence that it is natively built in *Caret*. \n",
    "\n",
    "The only thing you should do is to declare the parameter $\\texttt{tuneGrid}$ when invoking *Caret*'s $\\texttt{train}$ function.\n",
    "\n",
    "Note that:\n",
    "- $\\texttt{tuneGrid}$ is initialized as NULL\n",
    "- $\\texttt{tuneGrid}$ must be passed a named $\\texttt{data.frame}$ (plz read: \"a table with column names :)\")\n",
    "\n",
    "Finding which hyperparameters can be passed to our train function is farily easy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>parameter</th><th scope=col>class</th><th scope=col>label</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>kmax           </td><td>numeric        </td><td>Max. #Neighbors</td></tr>\n",
       "\t<tr><td>distance       </td><td>numeric        </td><td>Distance       </td></tr>\n",
       "\t<tr><td>kernel         </td><td>character      </td><td>Kernel         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " parameter & class & label\\\\\n",
       "\\hline\n",
       "\t kmax              & numeric           & Max. \\#Neighbors\\\\\n",
       "\t distance        & numeric         & Distance       \\\\\n",
       "\t kernel          & character       & Kernel         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "parameter | class | label | \n",
       "|---|---|---|\n",
       "| kmax            | numeric         | Max. #Neighbors | \n",
       "| distance        | numeric         | Distance        | \n",
       "| kernel          | character       | Kernel          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  parameter class     label          \n",
       "1 kmax      numeric   Max. #Neighbors\n",
       "2 distance  numeric   Distance       \n",
       "3 kernel    character Kernel         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "info = getModelInfo()\n",
    "info$kknn$parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw, *kknn* can be passed three parameters, two of which are of *numeric* type while the latter should be a *string*\n",
    "\n",
    "> if you decide to pass a tuning grid, you **must** declare a data.frame which would include **all** the parameters available. In our case, a data.frame with three columns (and, ofc, at least one row)\n",
    "\n",
    "While the concept of *kmax* argument may be of easy understanding, the same cannot be said of *distance* and *kernel*. As we saw in a previous lesson, to know more model-specific hyperparameters, you can refer to the reference manual of the given package, from which the model you are about to train is taken. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'kknn'"
      ],
      "text/latex": [
       "'kknn'"
      ],
      "text/markdown": [
       "'kknn'"
      ],
      "text/plain": [
       "[1] \"kknn\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "info$kknn$library ## as easy as that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And [here](https://cran.r-project.org/web/packages/kknn/kknn.pdf) you have it. It turns out that\n",
    "- $\\texttt{kmax}$ best number of k neighbours less or equal to k.\n",
    "- $\\texttt{distance}$ is the parameter of the [Minkowski distance](https://en.wikipedia.org/wiki/Minkowski_distance) which, as **everybody** knows, takes value in $(0;+\\infty)$\n",
    "- $\\texttt{kernel}$ is Kernel function to use. Possible choices are \"rectangular\" (which is standard unweighted knn), \"triangular\", \"epanechnikov\" (or beta(2,2)), \"biweight\" (or beta(3,3)), \"triweight\"\n",
    "(or beta(4,4)), \"cos\", \"inv\", \"gaussian\", \"rank\" and \"optimal\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=white-space:pre-wrap>'This tune grid has  60  rows!'</span>"
      ],
      "text/latex": [
       "'This tune grid has  60  rows!'"
      ],
      "text/markdown": [
       "<span style=white-space:pre-wrap>'This tune grid has  60  rows!'</span>"
      ],
      "text/plain": [
       "[1] \"This tune grid has  60  rows!\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## istantiate the tuning grid\n",
    "\n",
    "((tune_grid = expand.grid(kmax = 20, \n",
    "                        kernel = c('triangular',\n",
    "                                   'rectangular',\n",
    "                                   'epanechnikov',\n",
    "                                   'biweight',\n",
    "                                   'triweight',\n",
    "                                   'cos',\n",
    "                                   'inv',\n",
    "                                   'gaussian',\n",
    "                                   'rank',\n",
    "                                   'optimal'),\n",
    "                        distance = c(0.5, 0.1, 1, 2, 5, 10)\n",
    "                        )\n",
    ")\n",
    ") %>% nrow %>% paste('This tune grid has ', ., ' rows!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "k-Nearest Neighbors \n",
       "\n",
       "918 samples\n",
       "  8 predictor\n",
       "  2 classes: 'Not', 'Yes' \n",
       "\n",
       "No pre-processing\n",
       "Resampling: Cross-Validated (10 fold) \n",
       "Summary of sample sizes: 826, 826, 827, 826, 826, 826, ... \n",
       "Resampling results across tuning parameters:\n",
       "\n",
       "  kernel        distance  Accuracy   Kappa    \n",
       "  triangular     0.1      0.7810798  0.5370820\n",
       "  triangular     0.5      0.7822145  0.5393684\n",
       "  triangular     1.0      0.7953058  0.5660860\n",
       "  triangular     2.0      0.7952938  0.5661749\n",
       "  triangular     5.0      0.7876851  0.5520019\n",
       "  triangular    10.0      0.7843884  0.5444032\n",
       "  rectangular    0.1      0.7810798  0.5311686\n",
       "  rectangular    0.5      0.7811037  0.5339270\n",
       "  rectangular    1.0      0.7920091  0.5570212\n",
       "  rectangular    2.0      0.7985905  0.5740086\n",
       "  rectangular    5.0      0.7756928  0.5272391\n",
       "  rectangular   10.0      0.7757286  0.5262589\n",
       "  epanechnikov   0.1      0.7832418  0.5373270\n",
       "  epanechnikov   0.5      0.7788939  0.5319887\n",
       "  epanechnikov   1.0      0.7930960  0.5614046\n",
       "  epanechnikov   2.0      0.7920330  0.5597399\n",
       "  epanechnikov   5.0      0.7844004  0.5460632\n",
       "  epanechnikov  10.0      0.7844243  0.5453715\n",
       "  biweight       0.1      0.7821667  0.5390107\n",
       "  biweight       0.5      0.7854873  0.5454682\n",
       "  biweight       1.0      0.7985905  0.5737277\n",
       "  biweight       2.0      0.7920330  0.5606263\n",
       "  biweight       5.0      0.7843884  0.5461349\n",
       "  biweight      10.0      0.7822026  0.5409886\n",
       "  triweight      0.1      0.7887124  0.5512360\n",
       "  triweight      0.5      0.7822145  0.5392736\n",
       "  triweight      1.0      0.7930960  0.5629435\n",
       "  triweight      2.0      0.7865624  0.5506275\n",
       "  triweight      5.0      0.7811276  0.5383049\n",
       "  triweight     10.0      0.7756928  0.5275773\n",
       "  cos            0.1      0.7832656  0.5387050\n",
       "  cos            0.5      0.7799809  0.5345518\n",
       "  cos            1.0      0.7952938  0.5658154\n",
       "  cos            2.0      0.7920449  0.5603048\n",
       "  cos            5.0      0.7844243  0.5459824\n",
       "  cos           10.0      0.7887721  0.5546578\n",
       "  inv            0.1      0.7494983  0.4774317\n",
       "  inv            0.5      0.7712972  0.5214666\n",
       "  inv            1.0      0.7767678  0.5312433\n",
       "  inv            2.0      0.7745581  0.5289824\n",
       "  inv            5.0      0.7712852  0.5205695\n",
       "  inv           10.0      0.7734591  0.5252731\n",
       "  gaussian       0.1      0.7854396  0.5429992\n",
       "  gaussian       0.5      0.7810678  0.5352710\n",
       "  gaussian       1.0      0.7920330  0.5579441\n",
       "  gaussian       2.0      0.7920091  0.5603954\n",
       "  gaussian       5.0      0.7931199  0.5632046\n",
       "  gaussian      10.0      0.7887602  0.5546183\n",
       "  rank           0.1      0.7745342  0.5177869\n",
       "  rank           0.5      0.7778309  0.5299760\n",
       "  rank           1.0      0.7963808  0.5669591\n",
       "  rank           2.0      0.7930960  0.5609157\n",
       "  rank           5.0      0.7887721  0.5535314\n",
       "  rank          10.0      0.7887721  0.5532466\n",
       "  optimal        0.1      0.7778189  0.5278267\n",
       "  optimal        0.5      0.7909818  0.5590077\n",
       "  optimal        1.0      0.8029503  0.5831926\n",
       "  optimal        2.0      0.7985786  0.5743193\n",
       "  optimal        5.0      0.7887721  0.5525316\n",
       "  optimal       10.0      0.7974797  0.5725099\n",
       "\n",
       "Tuning parameter 'kmax' was held constant at a value of 20\n",
       "Accuracy was used to select the optimal model using the largest value.\n",
       "The final values used for the model were kmax = 20, distance = 1 and kernel\n",
       " = optimal."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## train the model (you really don't want to run this)\n",
    "\n",
    "set.seed(123) # to allow reproducibility of cross validation split\n",
    "\n",
    "#model_grid_search = train(x = titanic_df$x_train,\n",
    "#                          y = titanic_df$y_train$class,\n",
    "#                          method = 'kknn',\n",
    "#                          tuneGrid = tune_grid,\n",
    "#                          trControl = trainControl(method = 'cv',\n",
    "#                                                   p = 0.9,\n",
    "#                                                   number = 10)\n",
    "#                    )\n",
    "\n",
    "#saveRDS(model_grid_search, 'data/model_grid_search.R')\n",
    "\n",
    "\n",
    "# rather, we trained the model for you, how nice are we?!?\n",
    "\n",
    "(model_grid_search = readRDS('data/model_grid_search.R'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAFoCAMAAABJ+DwrAAAANlBMVEUAAAAaGhpNTU1oaGh8\nfHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PD///86YAbPAAAACXBIWXMA\nABJ0AAASdAHeZh94AAAXtElEQVR4nO2di3ajOgxFHdrp43Y6Lf//szfhaRvL2GAjWTlnrRke\nEZZy2DUGkmB6CFIkw10ABJUUgIZUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2pEoCGVKkQ\n0J35t8z/M52bwoz/nBW2ft9Ca0Myk14/qYjfN2NeEgqGlKoQ0B/mfZl/Nx9uij2gx+UcoO9I\nExF/Iq9BT6BCQNu9st1bDyn8HGGgUzRH/u0M0Ucb853aGKRRpcbQr+Zrmvvye8gKQPd/qW44\nvS1IpUrt/y/zZ5r7M6L9/nI/9v8dUqxDjq9X031MC0vAMIRYQr7fOtO9fU8bvnfm5WvNsuI6\nzv19vbfxNS7/vpg/ZmrLa+bx0mOLj7G1z/tk7OCdItdc9zLN22/vZoCaULEOrTMjAb/j4KMb\n2XqgsNL6Pqx7GxbWAAfoL2Nv+LrMT9W6QP83xr4Py3ea3xeg3WYeL90nH8Oqv2/D5NMvcs01\nNjO8DSsD1ISKAf0+jWo/h73/MZwYfgwDg4XWb2P+u7PWPRb8gDXk47f/vSP177Gm+9v//rFG\nFwvQw7jme+hdv1/Hv4rX3zXCa2Z46d7a16PMbpy8+DUsub4fgb+vj/dhZ4CaUDGgv6erZS/D\nWdmLWfFagJ6Y/+3MNmAJGTvDt8d07C9/rWHxPHv/oxioHNr4fYx27j2vFeE1Mw0qpta+5ji3\nhiXX21Dm7+P92BmgJlTuHOpl4ObvchX4++vj1QV6Aug+yt4GLCHjFZJ/j2YmfB2gZ70PwbPW\noHEabsbqwpcRu1uDzXnvZYCaULk99Z953B95e4wq+uHEawZh4W3G4tVsA7wQ+0xyC3T358ta\nCgAdacaebGrwr5nbGaAmVG5P/U5gDN3b533g+v7fvzDQj6kfkAq0U7vZzucAva1hC/QhKyA+\nFdxjj8557Kbn8UcfBPp3PK67AalDDjthZ91ESRly+JNtDcOkW4ccHW7TtKaCQD+Gzy/OudmX\nC/SfEY9P5yJd7wD9Pv1BTGdzVlv+7BA0BH8/Lk24QEea2XTFXz7Qb9P5YedmgJpQyWNqZ77m\nO+AvjysF4wW6lZ1P0/03Xtr1A/5NYfcu1byP19u+94H+Hs4Nv7vxsp0dEWnG6aHdIqfJ/U18\nL5ft1gxQEyoJ9OPGxfS5pM/5ZOqv3RmO9y4+pvHrEvBC3lgZSyTHtF/2jRUngm7GHUM7Rc6v\njjdWXr0MUBMqCfR9cLwMPz8fd57/fo2XiBde/ltvfVsB3y+Pjn0KcW99WxNvdtC/9/kutgc0\n3Yx3lcMucnn1817Qu58BakI4jYdUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2pEoCGVAlA\nQ6oEoCFVAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVQLQkCoBaEiV2gb6xl2Aft2c\niXw1DfStHZ+b1c36vwUBaCiq2/JfG1IB9O0uazrvg/vLN//F1jocdi2GWva608VhEWoZ6Jv1\nr78FpoEXAXSmbivP8+LNmy4vSpAaoP3pzTb55r4oxv0GdHMHdoF+o7en7GoY6KWz3QHaHmsA\n6FzdLMMsJx2gJY05WgZ61B7Q9oFxXBZjfguy+mFviHGz1snxtGWg5/+jQG/GHwA6Sw7L4VOV\nNUyA2gV6xTfk8M3pQtz+RIz5LWg9I1ycDJwUYshxXlZ/7F22Gy4nrUOOcWEd6Inxvgm5No+2\nbi/byTG1XaAPS4z3jUrchQ1HABrKFIAWJTmjvVYFoCHoMgFoSJUANKRKABpSJQANqRKAhlSp\nNNA/CD4azJhaUTCAFhPMmFpRMIAWE0w2Au1qdasQ0NxvqAWdNxnB+8HoocUEM6ZWFAygxQQz\nplYUDKDFBDOmVhQMoMUEM6ZWFAygxQQzplYUDKDFBDOmVhQMoMUEM6ZWFAygCwc7hgLoy70F\n0AC6mWAAzRAMoOsFA2iGYGM7CqAv9xZAA+hmggH09cGmB9C1gpO8BdAAupVgAM0QbBxLAfTl\n3iYB3d3lz9vr0mtSHwyg6wUXA7pb/lvn7XUZNakPBtD1ggE0QzCArhcMoBmCAXS94CuB5v6+\nnhiZ6V9ACU7H1QZ29YJNb/NaCOj5ZBA9dFhJpj/kn2h306Rbl2vW2WJwFaAx5Ii/nAq020l4\nL4QuH+2nVh8MoK8PNst/O8EU0JSv+6nVB6d5C6DLm76amgZ0YgfdhgP1gtO8BdACgF6HzJ23\nPDQCDTLW/778PbCj5e6gPV/gTmHWffefnA3aAnqd7/zX0lOrDy7XQ+cIQPfZQK8rrDVngX5W\nbwG0JKD9s8Oc1MFKEqXIW06gTb7pqVvINr2nhhz+uCMndbCSRAFoUgDamkTc9YG2RxzUlY7K\nQLfhrXEXiahyAtDWJOZu6ETbvuyRndor5Fm9BdAlg32zy7r73N5qBNoklwug1QGd6C0j0Ol4\nzi2b5Hplm35QADo0JcJK6TmApqrwzDZZLaelTtTTHv0ANICW5S2lRG8B9CGgd7oHAF0+uAWg\nf3KySzJ9D+j5mikb0A13FpQkAG0GEZH39RcBvbeNSqCb7SwoXQt05AN/xDfsYt++i32AMHOT\nWAXHZYg2jTvjBp03OT1UMdA7nUXdHjqW4VgPnViwd6DICE5qmTqit91DH/G2fPDeCTeAvg5o\n480BaI1ARy4YXgX0zkbZplOH5yuBjp6fXAf0Zecnqd4C6GNAB5v0TDd5LSel9jOEdaW3GcFJ\nLbcK9GPtEdPTKgbQ7Q7n5AJtnEkgcabp0Q6fLmP3BL4C0FNIY0Af8Zbp/IT09gmA3u17DgEd\navJyoAsP5yQc/RQCTZS0A7S7uiTQ25YBtJUJQK9rCaDDG5wCOvo+46abpwD6oLengXa2LjGc\n4wJ6WEn10EGkf/b/QCJlHAbaH4AvQAearAN07J4kcRM0/5Zq7FdcYk2ZczdizWZjqnQTDQnu\ngyIKAE3/wdFDjsALe0BvsHPLiLzRGHabYq4H2kntpbrm/KTacM6EvRXeQx8BOrDN1UCvdy5S\ngTbuvMkcQe6q/vnJMW/PHP2cjUsM5wA0CXRgrm8e6PAG8oH2vNUPdODg5ZZxAuiw6dsmjbtw\nBdCUGYXPT+oN5wA0gHZyZQMd3IQF6O226Ue/k0BbP3ziPDYhEHoeaMran732fGv2GgwGbzbz\nN20faAGdhb2RtXGJ4VwK0KEfFkz5/TU1QIdN3zSpF+gdb48AbTYzfECnPjQoYvq4SjLQhlj4\nCe+LwCZsQJ8Yzh3ztjzQx49+FYE2gTl3DYDOkligs49+BBxMQId+ZZ64ibS9y+OuidwSIrch\nt4p9eZHYKiJDL1FvzAvyEyY4HZee4RwxnuMGeudZegk9ND0aCZhOdYzzuoo9tGs69cZc0w3V\n8mEB6GWhPNCPqxznhxxFgfYPXnsNBoP9rTZLqUD31wBNOlgW6LLeJgB9eDiXD3TyYxOEAB3b\nVZvg7Wb+Eg10YBdwAp3lLYAuc1IYMX3z0k+svQSg6beaAPS66ABN7JhlsRrQu16IPvpRPUGJ\nE+7WgbYdTQN6m/IM0NZXrwF0QPvX+ClvI0e/s0A7j09If2zCMwAdK5hq+bCuADrv6Oe/90Rv\nTwMd6SySgM5Q0PRtlipAe6MxP3iKUQ90MW+dVSneXgV07OjXPNA2ygB6XVMO6PLeJgHtBEkH\n2szBZ003jt0FTScqlgd0xc7CWcMA9KHzE3FA+0extWWpQNPHWwB9Bmj7JQBNNRj6ljKABtCk\nwqYTdUaB9l/7Ce1EF2jHmnCDBYAmCMoE2r5KZH/GPO2z5tcDHfc2q7M4DzTdckWgnZYLAB38\n+7gM6GVFIaCd6/jeNOsaf2mgtzsAQIdaDi6VAHr20l1PAW1C77htoImDSRTopKNfDtBZnQV5\n+APQOoAOcl0DaNLbIkAne5sBdL63bQNtrH9ODAPQIScTgfaH0NZrj0Z2Pq3tf+7aEOtjn1A3\ngdn5E937n13/mT+PnviZ85Ts7vJew5Edd1IU0O5iLaDtv/VggzKBnv7zp3Rrsc6iWA+de37y\nXD00G9DOrjJ9qulXAt3b8yxAE8M5940V6ywA9GbR8dg4OyFSRhLQlIMRoJcy4i07qgd0KW9z\ngU7vLCJAU7ZfDvTRgVJsnJfwtb75QYiRrxs6o82EX+WkfoZzt+bd8aNjWMkhR53OYgY6dTin\nC+hAGqrqPvoGQmHjAtmPJvfQ452svR6a6G4r9tBd4F9loGN9vOXtznDOabAm0BneXgd0oH84\nC/TyfyrQwbpYgHY+Y77Me1OyNQDdJtDW56283zKjgF53Z6gMSUAfUNLR70qgc70N5u6j3rYG\ndHTX2Cv8H+czzmuJQIdXPCfQJrCNs7AB+nxnEcpmr2gM6ODf+i7Qy1sqBfR2zY/7YvjYqQzo\n3mG6ANDhFaeBzva2GtCBhk8AbTZvwPIaQAcGE/tAB3aIvTS/qXLekkHPBfQMWATo0O6kj1bH\ngF5apl4SB3RCBzobSB79AHQM6GVdEGhytBQG2vI6MBw8A7S7F/3NyF8k0gO09XYAdAWgpyHF\nFuilOQJowgsA7a3Y/Br+1UCvLasCmiw9DHQfPqfhAjpopHSgZ1zIo19hoHcOmq0CvbyUCPS4\nMhdo/86tX0Wg5vnF4IN1BAFN/7kRnUV8swjQvfOh/bWzoBp8MqD9N/Tj7IhcoLc7cZ3/WXYK\ngN7GuKtMrLMghnNUg3qBDrd7EGin5R0B6CVYINDhPSIfaOL5jnGgCf7clvc1d+eG4gxAu+tC\nQC+xRYGmPpInHmi6Ube82kD3oa9tJgBN7fohWBvQpo/VeS3Q1EvMQEfatD9vpAjocCINQPfc\nQJNUnAH64IM3d+UBTXZ0WS3bQAcHDzHTY3+LU7AuoBe7KJnNwhmg46PAEke/FKCPPnhzV1ug\nI+UA6OgJt/GmAJrW0Qdv7qoK0NMxdBo7KAQ6XONRoO2Wd7UATY1sAbS7H6irI1kt20Dnmt4A\n0ESJFwMdHwwTLQsFOvXBm0lyfxXFZD4dM9Ik/QslsRyR1+jfb/kJPWZzqwSng0qrMFBe5GvK\nB7V+GZn4unE0xfRd4shmmY9UXXUe6NInhbV6aOJ1quWYHSYScVkPTRYY7aFjpc8tp8g6P4l2\nteGWybMagT30Xpqw6gD9aOtyoMOrK13lCFbgTCoBPXUWuoBOfqzbvjZAFzkpfE6g/fJcoKmT\nuKSWvSTnrvED6PyWGYAmVgsButTnZBygs68gxQwcVyoEOh6cKADdby6JUliktOwluRjozPOT\nFKAPPnhzX+tZTkIZeUBHGgTQmS17WX4iViQATTXbXwh0hiQAfW8NQNe5aVX36AegCR0GOloG\ngO4B9H6wEqDDuhzotTof6IYviQLoikD73/iP60qgvWGydRdmshhAH5EIoO/YPSXQdvkboMsM\nOR6dBYCOBQPoUq25XbAioLPOT54caOO2vAv09hv/UV095PCC90lOa9nNctDbpwI6pYosoPfN\nswKVAL0NZgba9RZAn2m5BtDjYK49oPmOfiarsyBDAHQy0KbPMB1Aj80lAu0f0J4GaOd74KVa\nBtAigLbPoZ8F6OuDj/Yim0HhuTLy1SDQ1goAXSsYQJ9rOf3o1ycDHR15cgC9+6UvQfK/dRd7\npKa74bkv5Z03OTe4MaCpa/zoobN6aMvHHTP8C6vnysjXgfOTnu2Ee5pbxyfHgQ4LQM/amJ7o\neVmg7c+T+79G1YU/bH4gtYl+yO5My2HRQCd4C6CPBR8GumQZzjd+vGd+h74JdDh1+StIAFpa\nsA9wqukly7gM6GuDATRHsDSgO+8Fguc2vd0cBuntAPTBYHFAO+PpzRD61KWVq7X9DSTjLZPb\nnbmGtLr1zEDP0+WXHS8sY9NDW/OlvlHPEXzUW+PeEDheBoDmB7q35r1fi6iRumow4e2utQD6\nRPBR00uWQQDt/4JgjdRVg7fe9kkXwwH0iWBpQPu/eaJryDGyuicAfSL4qOkly4iNm6krdw15\nu5gJoC8IlgC082tU1o3Cks+v4QgOAp1gbbGP5j4x0NmmFy4jW416m3SzEkCfCD5qeuEystWs\ntymtGgB9OPiw6WXLyJZqby8FOvCcwq7cJ8IuDw6ZzlBGtoTUWcfbK4GmbgI0fmkJQFcIbhLo\nCM+6Ta8bzJhagLemUBkAWkzNjKkFeMsIdIjnEx+Uul70IwyrKmdvBQWgE8ooBPRemqyaagej\nh64XDKAZggF0vWD/c4uXl5EPdJTnhkwH0DWCATRDMLvpByWkTtneAmgxNTOmVuRt1p3C0FeV\nK9RUO5jd9IMSUqdsb5/3sxwAukYwu7cAWkzNjKkVeQugxdTMmFqRtwBaTM2MqRV5C6DF1Ew2\n0pLSflamuFa3ALSYmhlTK/IWQIupmTG1Im8BtJiaGVMr8hZAi6mZMbUibwG0mJoZUyvyFkCL\nqZkxtSJvAbSYmhlTK/IWQIupmTG1Im8LAX31hfRTYr/4f9TkBoLN8h9TGeihxdTMmBpA56YR\nFcxu+kEJqVO2twBaTM2MqRV5+4xAD28aQNcJ5vYWQIupmTG1Im8BtJiaGVMr8hZAi6mZMbUi\nbwG0mJoZUyvyFkCLqZkxtSJvAbSYmhlTK/IWQIupmTG1Im8BtJiaGVMr8hZAM5VhP3PJf+Bm\nww/eZPcWQPOUEfqZwHldy8/6ZvcWQANoVd4mAR14TiEOi+eC/YfV2+sA9IngFKBDzymE6eeC\nHaDnPmJa53l79RcRzsn8MHx3wvL5CNBdYN3+LgwIQDvzWnpom6kmeuiuD/pdrKbqwdym9/Gj\nHoA+EXwA6GnRGUNffow5JcNxVPyJeQqgSwUfBpropmF6WjCGHJXKON5D9zD9RLAPdOSksHTq\nusHc3uYD3QXWla2pejC36Q/ZD2Ly7xA2fkkUQF8dzG36QQmpU7a3AFpMzYypFXmbdafQvxtQ\nqabqwdymH5SQOmV7+6yf5QDQlYK5vQXQYmpmTK3IWwAtpmbG1Iq8BdBiamZMrchbAC2mZsbU\nirwF0GJqZkytyFsALaZmxtSKvAXQYmpmTK3IWwAtpmbG1Iq8BdBiamZMXdRbBykAfUUwt+kH\nJaRO2d4CaDE1M6ZW5C2AFlMzY2pF3gJoMTUzplbkbSGgr//K6RkZju/InocbQCcEo4cWUzPZ\nSFPi7iwAtJiaGVMr8vYpge4NgK4WzOwtgBZTM2NqRd4CaDE1M6ZW5C2AFlMzY2pF3gJoMTUz\nplbkLYAWUzNjakXeAmgxNTOmVuQtgBZTM2NqRd4CaDE1M6ZW5C2AFlMzY2pF3gJoMTUzpi7q\nLW8ZTwq0jDIyJaRO2d4CaDE1M6ZW5G3Wz+n23oM3W/19aG7TD0pInbK9zf/B8z76XDeYfjiY\nMbUib48ATf16f6ma6gcD6HrBjQLd9IM3uU0/KCF1yvb2ANDLg5qaffAmy7eEzsMNoBOCDwPt\nritaU/1g9ND1ghsF2psrWlP9YABdL7g5oP2zw/I1PUWwd9VzuRLa/CVR5uCjQLc95OAP9o96\njr1NX0FiDj4DdLPPKRQQ7HvaBddXSa07OOtOoZYHbwoIdsDtnNnGL4kyBz/nZzn4gwmgFVwS\nZdHqFoDmB7rrPaCt1yqk1h0MoNmB9tn250qn1h0MoPmBXi7X4ZLo+WAAzQ70uoxLoueDAbRM\noHEF6WBwaaChRHmjDBto6pIolCAADakSgIZUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2p\nUlmg69ziWm6q1Wi9Usvla4a3S7OxmosCHfv+0PlWq7TeLR+cKNpy+Zrh7dLs9F+45ScHuusB\ntC5vGwB6brpC611f0/SiLcNbq+F1CqBDrdYwvSt8wIW3a8MxbxsBut7feqWWp2ZbAFqXt88M\n9NLg846h50b1eNsG0J37X6lW5y/zPTXQyrxtAuiuYuvP3kNr87YFoLuarT850Oq8beBOofOj\nnOVbn1MUbraNO4X6vMVnOSBVAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVQLQkCoB\naEiVADSkSgAaUiUADakSgIZUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2pEoCGVAlAQ6oE\noCFVAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVfofZQJnwmKDOIAAAAAASUVORK5C\nYII=",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "options(repr.plot.width=6, repr.plot.height=3)\n",
    "\n",
    "model_grid_search$results %>%\n",
    "  as_data_frame %>%\n",
    "  rownames_to_column(var = 'id') %>% mutate(id = as.numeric(id))%>%\n",
    "    gather(Variable, Value, -one_of('id', 'kmax', 'kernel', 'distance')) %>%\n",
    "    filter(Variable == 'Accuracy' | Variable == 'Kappa') %>%\n",
    "    ggplot() + geom_line(aes(x = id, y = Value)) + ggtitle('Validation Performance') +\n",
    "    facet_wrap(~Variable, scales = 'free') + theme_minimal() + labs(x = \"\", y = \"\")\n",
    "\n",
    "options(repr.plot.width=6, repr.plot.height=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like there is a clear winner here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>kmax</th><th scope=col>kernel</th><th scope=col>distance</th><th scope=col>Accuracy</th><th scope=col>Kappa</th><th scope=col>AccuracySD</th><th scope=col>KappaSD</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>20        </td><td>optimal   </td><td>1         </td><td>0.8029503 </td><td>0.5831926 </td><td>0.05869298</td><td>0.1213537 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       " kmax & kernel & distance & Accuracy & Kappa & AccuracySD & KappaSD\\\\\n",
       "\\hline\n",
       "\t 20         & optimal    & 1          & 0.8029503  & 0.5831926  & 0.05869298 & 0.1213537 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "kmax | kernel | distance | Accuracy | Kappa | AccuracySD | KappaSD | \n",
       "|---|\n",
       "| 20         | optimal    | 1          | 0.8029503  | 0.5831926  | 0.05869298 | 0.1213537  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  kmax kernel  distance Accuracy  Kappa     AccuracySD KappaSD  \n",
       "1 20   optimal 1        0.8029503 0.5831926 0.05869298 0.1213537"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>kmax</th><th scope=col>kernel</th><th scope=col>distance</th><th scope=col>Accuracy</th><th scope=col>Kappa</th><th scope=col>AccuracySD</th><th scope=col>KappaSD</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>20        </td><td>optimal   </td><td>1         </td><td>0.8029503 </td><td>0.5831926 </td><td>0.05869298</td><td>0.1213537 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       " kmax & kernel & distance & Accuracy & Kappa & AccuracySD & KappaSD\\\\\n",
       "\\hline\n",
       "\t 20         & optimal    & 1          & 0.8029503  & 0.5831926  & 0.05869298 & 0.1213537 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "kmax | kernel | distance | Accuracy | Kappa | AccuracySD | KappaSD | \n",
       "|---|\n",
       "| 20         | optimal    | 1          | 0.8029503  | 0.5831926  | 0.05869298 | 0.1213537  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  kmax kernel  distance Accuracy  Kappa     AccuracySD KappaSD  \n",
       "1 20   optimal 1        0.8029503 0.5831926 0.05869298 0.1213537"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_grid_search$results %>% as_data_frame %>% top_n(1, Accuracy)\n",
    "model_grid_search$results %>% as_data_frame %>% top_n(1, Kappa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>kmax</th><th scope=col>distance</th><th scope=col>kernel</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>57</th><td>20     </td><td>1      </td><td>optimal</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       "  & kmax & distance & kernel\\\\\n",
       "\\hline\n",
       "\t57 & 20      & 1       & optimal\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | kmax | distance | kernel | \n",
       "|---|\n",
       "| 57 | 20      | 1       | optimal | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "   kmax distance kernel \n",
       "57 20   1        optimal"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_grid_search$bestTune # indeed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computational Issues\n",
    "\n",
    "As you see, (if you tried training the model yourself) the model takes a **looong** time to train\n",
    "\n",
    "Indeed, grid search is often not viable when the hyperparameter space gets lerger, indeed, it gets exponentially complex."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manual Search\n",
    "\n",
    "This may seem trivial, but it actually requires a good undertstanding of the underlying problem + it can help in reducing the space of grid search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Search\n",
    "\n",
    "grid search looked promising but is there a better way to do it?\n",
    "\n",
    "Randomly sampling in the hyperparameter space may yield good results with less computing time. Also, it potentially allows us to explore an even larger hyperparameter space!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "k-Nearest Neighbors \n",
       "\n",
       "918 samples\n",
       "  8 predictor\n",
       "  2 classes: 'Not', 'Yes' \n",
       "\n",
       "No pre-processing\n",
       "Resampling: Cross-Validated (10 fold) \n",
       "Summary of sample sizes: 826, 828, 826, 826, 826, 826, ... \n",
       "Resampling results across tuning parameters:\n",
       "\n",
       "  kernel        distance  Accuracy   Kappa    \n",
       "  rectangular   10.0      0.7875121  0.5480525\n",
       "  epanechnikov   0.1      0.7941787  0.5582882\n",
       "  epanechnikov  10.0      0.7854348  0.5451215\n",
       "  biweight       1.0      0.8017633  0.5813310\n",
       "  biweight      10.0      0.7778261  0.5296114\n",
       "  inv            5.0      0.7854589  0.5496455\n",
       "  gaussian       0.5      0.7962560  0.5667048\n",
       "  gaussian       5.0      0.7810386  0.5347831\n",
       "  gaussian      10.0      0.7885990  0.5506830\n",
       "  rank           1.0      0.7930435  0.5585456\n",
       "\n",
       "Tuning parameter 'kmax' was held constant at a value of 20\n",
       "Accuracy was used to select the optimal model using the largest value.\n",
       "The final values used for the model were kmax = 20, distance = 1 and kernel\n",
       " = biweight."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## train the model (you really don't want to run this)\n",
    "\n",
    "set.seed(123) # to allow reproducibility of cross validation split\n",
    "\n",
    "tune_grid_random = tune_grid %>% sample_n(10)\n",
    "\n",
    "#model_random_search = train(x = titanic_df$x_train,\n",
    "#                          y = titanic_df$y_train$class,\n",
    "#                          method = 'kknn',\n",
    "#                          tuneGrid = tune_grid_random,\n",
    "#                          trControl = trainControl(method = 'cv',\n",
    "#                                                   p = 0.9,\n",
    "#                                                   number = 10)\n",
    "#                    )\n",
    "\n",
    "#saveRDS(model_random_search, 'data/model_random_search.R')\n",
    "\n",
    "\n",
    "# rather we trained the model for you, how nice are we?!?\n",
    "\n",
    "(model_random_search = readRDS('data/model_random_search.R'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAFoCAMAAABJ+DwrAAAANlBMVEUAAAAaGhpNTU1oaGh8\nfHyMjIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enr6+vw8PD///86YAbPAAAACXBIWXMA\nABJ0AAASdAHeZh94AAAXnUlEQVR4nO2di3riOAyFTdrpZdrOlvd/2QUCiZ34ItuyI4tz9tsh\nwEEy4q/xBYg5Q5AimaMbAEGcAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVQLQkCox\nAz2Zf8vxPzO5qcz8v3ODrd83360+mbteP0OO3zdjXggNhpSJGegP874cv5sPN1UK6Pl6DtAX\npAOOP5H7IMViBtrule3e+pZqm8sPNEUP5/dkAn20MT/UYJAmcY+hX83X/ehr20M2APr8HeqG\n6bEgVeJ+3b/Mn/vRnxnt95fLe//3LdU65Ph6NdPH/cpiuA0hFsvP22Smt5/7A98n8/K1Zllx\nnY++Xy8xvubrvy/mj7nH2oS53nV9xMcc7fNyMXfwTiPXXJdmmrffs5sBEi32jmwyMwG/8+Bj\nmtm6orDS+n677e12ZTU4QH8Z+4Gvy/G91S7Qf2fv++36heb3BWg3zPWuy8XH7abvt9vF57aR\na645zO1pWBkg0WIH+v0+qv28vfoft4nhx21gsND6Y8zfC2vT9crWsFo+fs+/F6T+XW+Zvs+/\nf6zRxQL0bVzzc+tdf17nv4rX39WxCXO76xLt69rMab542bZhyfVzNf6+Xp+HnQESLXagf+6r\nZS+3WdmLWfFagL4z/zuZvWGxzJ3h2/Vy7i9/rWHx4/DyR3Gj8hbj9zraufS8lmMT5j6ouEf7\nevjcNiy53m7N/L0+HzsDJFr8c6eXGzffyyrwz9fHqwv0HaDLKHtvWCzzCsm/a5g7vg7QD73f\nzA+tpvnSH8bqwpcRu9sGm/PzJgMkWvyv0F9z3R95u44qzreJ1wOEhbcHFq9mb9hY7JnkHujp\nz5d1zQN0JIx9sWvDds3czgCJFv8r9HsH49a9fV4Gru9///mBvl5uDVSgnedg9sc5QO/bsAe6\nqBRQfzV4pa6d89xNP8YfZy/Qv/P7umugDjnshJO1iUIZcmwv9m24XUzrkGPCNs0oagD0dfj8\n4szNvlyg/8x4fDqLdGcH6Pf7H8R9NmfF2h7eTDfzz3VpwgU6EmbXFX9tgX67zw8nNwMkWi3e\nSyfz9dgBf7muFMwLdCs7n2b6Oy/tbg3/7rZLl2re5/W2nzTQP7e54c80L9vZjkgYp4d2G3m/\nuDyJn2XZbs0AiVYLoK8bF/fPJX0+JlPfdmc471183Mevi+EluLEyNzU4pv2yN1YcRziMO4Z2\nGvm4d95Yed1kgESrBdCXwfEy/Py87jx/f81LxAsvf9etb8vw83Lt2O8Wd+vbutgc3vTv/bGL\nvQE6HGazymE3crn389Kg920GSLQwfYdUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2pEoCG\nVAlAQ6oEoCFVAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVQLQkCoBaEiVdAB9OroB\nenVyLuRLBdCnceo9nE7WvyMIQENRnZZ/xpAqoE8XWZeP1+Jy92l752gdz2FaCmmV1b1cKitC\nGoA+Wf+fT55Lz50AmqjTyvPj6mlzudwpQeqA3l6e7GKf3DvFvAqCdXIHdJ7+4mxfHi4FQC+d\nbQJoe6wBoKk6WYWyKugALWnMoQHoWSmg7TfI+bqYF0GyrH54M8Q4WbfJqaUGoB//RoHejT8A\nNEkOy/4pymoToPGBXvH1VfrkdCVuvyLmRZCsdUa4VNAzKcSQg09Wf7xZtrstK61DjvnKOuAT\n8xqIllveuZz7ZTs5xRwf6GKJeQ0Gk7iFDUcAGsoUgBYpOaO+0QSgIaibADSkSgAaUiUADakS\ngIZUCUBDqtQD6P80m/iyZWnUJ9ncBKArTQBalglAV5oAtCwTgK40AWhZJmag/4PSYil0V5MM\nVkkm9NCVJvTQskwAutIEoGWZAHSlCUDLMgHoShOAlmUC0JUmAC3LBKArTQBalmlIoOONHvW1\nzlLfmspglWQC0JUmAC3LBKArTbWBpovs42lzW6O0OxOApov7SRhFQE/LP+ulc1ubtDtTvKYy\nWCWZhgRaUneiBehoTWWwSjIB6EoTI9CT57ZGaXcmAJ0hAE0E+jGEtm67PriHzOW/gbUWFEBX\nmrh76IMmheihM8T8JIyoKTnnGPp+fMCQI1FTGaySTAC60gSgZZkAdKWpwZADQFeYxgRaUPWZ\ngfZMCluk3ZqeDOjtbpbvslH7PCZVQK91PAuuqQxWSSYK0LG3xf69ya3FeoAuVN+aymCVZALQ\nlSYALcsEoCtNAFqWiRnoDptCZvlnVFFerpQANBPQ5832LHpotmxZAtBahhxzgwE0pylZUxms\nkkyDAi2n+nqAjtRUBqskE4CuNAFoWSYAXWkC0LJMWTuFvt2s3rtaAJo57TMCXSnOJ/FoL4Dm\nM6VrKoNVkglAV5oAtCwTgK40AWhZplGBFlP99kA337k0m8sBtVYLQFeaFPXQ4ZrK6HxJpsGA\nXpsLoLlMhJrKYJVkAtCVJgAtywSgK00AWpYJQFeaALQsE4CuNAFoWaaxgDaB40bpngJo4z1s\nlQ1AOwLQ/GkpNZXBKskEoCtNAFqWCUBXmgC0LBOArjQBaFkmZqAbb9mbwPFYYik0l8nYJgBN\nUt/io4fOMgHofLUpvpDqA2hZJgBdaVIFdKimMlglmQB0pQlAyzIB6EpTbSDPiTenKf3NY672\nmzOAzlbf4o8FtPsza9s7mqVdBKALBKDzge710xAAukAAmgZ0RgcNoBUCLaP6nECvQ2eb56ab\nPCZ6dRit1QLQlSbuHno6W7c0S/uQOaOHzlej4suoPucYej0+CuhATWWwSjIB6EjEw4BO8gyg\nATTFdCzQ0/aGdmkfAtAlGgZoczzQxBEHgAbQBJPZhOwAtPenirsBbXYmAF2eOtNELf5gQBcK\nQCsEmrv6ZhsRQLNm62IC0E64pwfaX1MZrJJMJKCtD38tnwSzT0nR5YM0AJo/7ZMC7VsrXScy\nxakzTV2AdkNqB9p4TM8ItH0m2W5Ak4tfnM7sQgJozmx9TLVAE3gG0AC6n6kAaGdDazOEbvd5\nKt/nwHg/G2b4Q/pFeblSGgbogr2qw4Ceep5401d81uqbfUj00PXZRgH62jV7vzpET51nAtDs\naY3PxAx0yacJOgPt+Z5QB6C9xQfQVSZ6TSuALlgJPQTo3uf6bg602VxSIwHomKloaf9woHt8\n/w1A86d9WqCdT4TZX4CzLxu176FuQNsxAXRtNqFAV4pj+tsaaOM5EgJ0q9XDwAol58LlASuh\ngwBtAia27kQw0K3SBmrK2UOX7VWN2UPnTX8BNH/a9kCbfUAA/XD7TQC63JRRUwCdTJ35ybbW\nQBvvMYCuywaggwLQ7GlNyMQGdOlK6IhAZ84WcooPoGkmAF2sEND0Z9oaaOO/AqBrsvlW9rUC\nnfu5iWDxmapv/NcAdE02AB0WgOZP2xpo4z3UCXT2NnNjoDdBADRHNgAdVoSwMYDmr3I1GFk1\nBdDR1N6nqhloI7CHbg20f90IQLuuNkDvYoRHOGXZnh3ozIWj0YDOX/MdHOjtmQc4JB3owDRb\nPdC0t/eYiaH6jYE2IieFAJokwkdXTeQa5RE5d5K0j8D6Cd59MJZCV5piiAHorNT5a77RctRX\nfx+BtYc2cZPnTLI9vgXUGOjQNFsh0NuEBHi6A/24iQNoEzf5TrzZ44vHeTWtHsVlLRwB6Eg6\nQpuaAm0SJp1AB2cl+oAOvr8fBrSvAmxAJwP5ziTb4/cCAXSFKoGOL8trAvoxhD43/73A+LS3\nelK8C9D6i7K7ijfV8wKdHk/5Tus2EX7spLKny6xpfUlz5tkj9dBhekqLX1t9fwFSbaJlI7zR\nRn9MvjAtwQSgawSgnwvoyHueNqC92dQCTVl99Q45rNsK0lJMALpGCaATS7X2Y/iBDjx/DqAT\nf4izfGeSbQ90bk0BtD91jB5ZQKf2QwjZUu2+y3cm2eY7hU2Bjr3nPQfQ8d1h5UAXyh/RUExn\nFUCHPpLbE+hgruOAjjapCmhDMZXLG5H8a1QtgY4O4tiADn4kd3ygW1S/HmgiWcV6cqCNhCFH\nJFW4fYSeTiLQ1Pf+YoUwJL0xJE0VJY1XVNUYeiig4+8ayWxHAG12idUCHXlx+gEdzRT+llJT\noON/Y+VAFzUpS8HVY8pcNLumwoCOvX2SgE6dvJ7UvicCumyemqV9RLPPfQDQ8bc8HqCjkShA\nx7ZnyZsA8UQAOlO7iGZ3oBXo+PSyBOjdub7rgSYuKzIDHWtTBdD7sO2BNp6jihWYUqBTFWUA\n2sRNvYBO5akAukV3co5OVOPZPFF7Am0dagTaJEwFQLsnr3fvC34CO/UJ79D91d8KL35gadT0\n4wglT2oTxHiPFQKd+eT8Yjh5fTpNfD1+jZR+KEv1y3toX9TWPbTxX+sOdOotT8g6dOzk9QA6\n3piMJmVpQKDJ05KIKXeC4Ffs5PVsQP/ntZAKKwlof8zGQG+TxudNBTUlljQ18adHikRImEqB\nzpsUErL0Bzo5TxUCdP6oneErsEUziMSD+L97uyj1ArgC0PFA4aZkNSlLdsR90uhS7bA9NKlN\nFKCrT15PSeIvB43Vouq3ADoUsinQnqTHAJ18mYmjuKp+iwR0pZ4G6GDElkBH3tlUAU3cP+sB\nNClHd6AJCy8jAO3PGQGaBkZBSUkrWcVA76LLBzo8FrRN6QSHAR2O2B3oyKLvmEDvgx8HNO2E\nDOMDHQnYDuhg0uAMrKSmxwNNNkkCet+YhkATnnge0LGAzYAOJ+0ONKVaxJNz7E2e53kY0BlP\nAkCTlQQ62B2OCLTvaWoGuuD9kbTyQludIXhbAR1LGgC6aHLFxWoh0N6neRTQWeOmTWuoxW8y\n4COajgQ63j4/PQMCnbX7CqArTen3/vMzAE18menzqUBLQqaUm03BuclWvvdQAB2NmNwZUgF0\n6FkCaOcBbNSnB7PnRkCnN+ZqgHaMBwKdu1nVGOjw+v5WEoGmzBwPA5qyTEO5iQVoar9VMuTI\nMwkD2mlPWfGPADrlA9APXy6r2buvbYE2sdSuugJNnW0LBpo0HErvvfIATe+3SLytkSJ21UBn\nDvi4gU7aItlCZ5JNfCaXBsboQBdsVjUF2kRTu/Ks5x4PNKE8lUD7TrxJOVEhEYzkVpVooEvW\n9pmBrv2SjfM1m4yv7GR+u4dspxkzs9sFKwSaOhbdvryel5sB6Iw34tGAzkkdMJndASFSkzUm\nag9N68b9KjuTLH3HLbVV1Rto4pc9di0ImvxtaSsATQTaP4Sueksw0as5D2UwZpozrGu1AHS0\nTRRWyYvVPhWd1o3e/uTKPq2m8WxZU6XWm1XygF6a1AzonN1LwnyPDeizfRwBOm/31WkegOZQ\nB6Dp1T9zA50xKPSoAOjMzar4QigD0IZiWiKRy1W4ti8X6KziqwCaOOTIBDq+ENodaPIbmh6g\n721qBnQWEMn6ZG59bRU6k2yCZ/1Al67tA+h4m1oDnX0m2YLNqrWJLYA2FJMV6QmBnhslA+hk\ngWqBzlU/oC13f6CLl0KfD+jMt+xEgRg/D5mlrLSRaXY10DTqbVPbtX2xQBcXH0DvTACaV/mv\nuckuPr2wrEAX/9BErfLSRhaOZAJdvrYPoBNtAtBED4COmBoCnT2pilXIlDw5FhWt7Q8DdMXa\nvkygL80C0PURN0D7n0l3oNtuVkkFOrf4ADpuCs6z64C2Q/IBXbMUKhTo7OJTgc7c1gq2ZAmm\nGGjCEvOgQKdOXk85NXJm++QAHSnRQEAHpyUCga5aCqUAHTt5PeX7b3yvecxE27HiBDpvy5FX\nMoB2IvKM4s4HAG2f63swoHP3aZ3HeIONAnRoWiIP6Lql0FqgCTxrBTpzQZtXAJoPaPfk9cnv\nv/VSs9Pchx5UfB5JysuVUtFSaG6kFNBuxLGBfkwKiadGTqtLD52/T+s+yhOrfQ9d+ndTr9Sf\nbPGfdOyBRUF3L0tUsR767N7nf0VIL1utiQB0yS7A+XCgyyOy1dTv2dSFZeGoem0/H+hpfxuA\npgVilFqga5dCa4GWM+QgrMhxAl2w/scpAN0Q6OS5vgk5egBdtAtwBtABz7YsLEBXL4Vm7RSW\nnry+on0ZpmZA+4pUsv7HqbGBbri2L/WzHAWmFNCFi6bbR25uAtAl2dqt7QNoQrp9kcrW/xh1\nPNC7ogBoZhOAZjf1Bpph5eh5gC7d1to91L1BNdAxwqqAbrcUCqAp6bZVAtAecAA0tylajvJF\n0/2DAXQboDmWQgE0Jd3mwcXrf3wC0E8OdM0aE4DeezzcVAPNshQKoEnpnIeXL5fwaXSgm63t\nA2hSOgDtvb0iW6ul0CcBum6NyX18zXIJm/rWFEBTUrObhgTadybZs5gPfHUEmmnlSBPQ4Y3T\nyim5E6BquWSnwOfKpXzgK1BSLzU1o7gzgPapHdBWhB5AT8J76AZAc60cPQXQ1VPyPkDbHzkH\n0PRIwTitpAroyuWSrXxnkt0AXfhFVB55v7Ja/OXYSJC6mJ4XikdVraoW+w8OeEIwxAoCbcEs\nuof2Q1PVQ7MthT5DD10/JV9j1C6XbOX5mTXK1zQVAN1obR9AE9OZQKwGQE+NfgCzyGT2ngAz\nNUDzLYWqAtpXfZ4ZzCNK/XLJRr4hh3VREJHZBKDJqRuYxgea+FMnyoBmXDkC0MR0jYD2nkn2\nLBroEDKlo7h9SAD9kA9onhnMHIdh/Y9JqoDmXAoF0NR0ANq6gSUbgC4BmmnAdwvEslzCI01A\ns64cAWhquicF+vak2wDdZClUF9C76vMN+M75p07MNGXpMKDDwADoBqaGQAfO6QCgy7O1WDkC\n0OR0XLNLFgHo5wSaccAHoB/X2LKZQEQAvagl0I0D5UkF0C2WQnUDzfn+2DpQntQAzb1yBKBr\n0nEGylPfmv5nHXNmA9BJhwHQ3KamQLOvHJGATp28vix1E5MDNO/7Y+NAedIBdIOlUArQsZPX\nCzoL1k0Kga7/zledjPeQPXaV1mqVAL0717dQoLm7k7aB8nRMDx2lpaiHZosUj+gKQPcIlKdD\ngI7D0vNlDpsKgHZPXu/ex/QOUq6W749corxcKQFoZqDlnbx+Vsv3x7aB8gSg+/TQOambmAA0\nv8ncPQlWRgVa7Lm+Z5nO3QmAZs5WaQLQlSYAzZyt0qQW6F7Vfx6gU6iMA/QgJ6+fBaDZTfqA\nrtQRQHerPoBmzlZpAtCVpqcBOkkKgG5iAtD8JgOgKakbmTpXH0BzZwPQrgA0u+la0jQoALqN\nqXP1ATR3NgDtCkCzmy4lJXACoNuYBgPad+JNcWv7AJqSupGpc/UrA/nOUyhv9xVAU1I3MgFo\ndpP5j4IJgG5k8n/xslU6RqCn3R3N0maa+pYUQLsaF2hnPL0++HgZsd/+uWstqEagu6bj7qEf\nn/hqmzbT1Lek6KFdDQv02T6WNeTomg1AH2l6CqAF1h1ANzI1GHKIW+WQWPdOQB89ORhCdsF8\nJ94E0BUm9NCVptpAvhNvStsplFh3AN3IxJctS6M+SQAt3QSgZZkAdKUJQMsyAehKE4CWZQLQ\nlSYALcsEoCtNAFqWqQfQENRNABpSJQANqRKAhlQJQEOqBKAhVQLQkCoBaEiVADSkSgAaUqXD\ngfb9FFZ5pMk6rgtkRyoM5ZzQsb5ZGYm5aiqupOmaHg2075eDakJ5otZFq3kRt01haBYpsZ2m\nKp20khJqCqATwcp7tzOA9ger6CPSNT0a6Jt8P4VVGmQ9Zql+TZCjgF5z8fEspKSDAc03hLai\n1rSIYbx3NNBsQ2graE2Dapu0bYtEoHlqxkqO+/ZYMyk8CGiWmsor6VhA76+UhuKrvudaXpCD\ngd5fKYwko6RDAM3zTJ1H11Z/il7NinIM0Iw1FVXSEYCe3CMRQw6eNh0HNFNN5ZV0AKAn97C8\nZPajeavfrPitxFVTeSWVD/SyhTTdr9SFOrNEsovFtqvF0CxqXraaiitpuqZHAw1BrALQkCoB\naEiVADSkSgAaUiUADakSgIZUCUBDqgSgIVUC0JAqAWhIlQA0pEoAGlIlAA2pEoCGVAlAQ6oE\noCFVAtCQKgFoSJUANKRKABpSJQANqRKAhlQJQEOqBKAhVQLQkCoBaEiVADSkSgAaUiUADakS\ngIZUCUBDqgSgIVX6Hxf+0n07snFYAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "options(repr.plot.width=6, repr.plot.height=3)\n",
    "\n",
    "model_random_search$results %>%\n",
    "  as_data_frame %>%\n",
    "  rownames_to_column(var = 'id') %>% mutate(id = as.numeric(id))%>%\n",
    "    gather(Variable, Value, -one_of('id', 'kmax', 'kernel', 'distance')) %>%\n",
    "    filter(Variable == 'Accuracy' | Variable == 'Kappa') %>%\n",
    "    ggplot() + geom_line(aes(x = id, y = Value)) + ggtitle('Validation Performance') +\n",
    "    facet_wrap(~Variable, scales = 'free') + theme_minimal() + labs(x = \"\", y = \"\")\n",
    "\n",
    "options(repr.plot.width=6, repr.plot.height=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>kmax</th><th scope=col>kernel</th><th scope=col>distance</th><th scope=col>Accuracy</th><th scope=col>Kappa</th><th scope=col>AccuracySD</th><th scope=col>KappaSD</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>20        </td><td>biweight  </td><td>1         </td><td>0.8017633 </td><td>0.581331  </td><td>0.05057195</td><td>0.1055897 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       " kmax & kernel & distance & Accuracy & Kappa & AccuracySD & KappaSD\\\\\n",
       "\\hline\n",
       "\t 20         & biweight   & 1          & 0.8017633  & 0.581331   & 0.05057195 & 0.1055897 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "kmax | kernel | distance | Accuracy | Kappa | AccuracySD | KappaSD | \n",
       "|---|\n",
       "| 20         | biweight   | 1          | 0.8017633  | 0.581331   | 0.05057195 | 0.1055897  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  kmax kernel   distance Accuracy  Kappa    AccuracySD KappaSD  \n",
       "1 20   biweight 1        0.8017633 0.581331 0.05057195 0.1055897"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>kmax</th><th scope=col>kernel</th><th scope=col>distance</th><th scope=col>Accuracy</th><th scope=col>Kappa</th><th scope=col>AccuracySD</th><th scope=col>KappaSD</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>20        </td><td>biweight  </td><td>1         </td><td>0.8017633 </td><td>0.581331  </td><td>0.05057195</td><td>0.1055897 </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllll}\n",
       " kmax & kernel & distance & Accuracy & Kappa & AccuracySD & KappaSD\\\\\n",
       "\\hline\n",
       "\t 20         & biweight   & 1          & 0.8017633  & 0.581331   & 0.05057195 & 0.1055897 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "kmax | kernel | distance | Accuracy | Kappa | AccuracySD | KappaSD | \n",
       "|---|\n",
       "| 20         | biweight   | 1          | 0.8017633  | 0.581331   | 0.05057195 | 0.1055897  | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  kmax kernel   distance Accuracy  Kappa    AccuracySD KappaSD  \n",
       "1 20   biweight 1        0.8017633 0.581331 0.05057195 0.1055897"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_random_search$results %>% as_data_frame %>% top_n(1, Accuracy)\n",
    "model_random_search$results %>% as_data_frame %>% top_n(1, Kappa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! with the 1/6th of training time we found a solution almost as good as than with grid search!\n",
    "- note that despite this is not be the *best* hyperparameter set, it does they job fairly good, don't you think?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayesian Optimization\n",
    "\n",
    "[Google says](https://cloud.google.com/blog/big-data/2017/08/hyperparameter-tuning-in-cloud-machine-learning-engine-using-bayesian-optimization) it works the best.\n",
    "\n",
    "This is a greedy approach (not a grid approach ;) ) in the sense that it is aimed at quickly maximize potentially locally optimal solution. Shoudl be relatively faster to other approaches if the hyperparameter space is particularly large! \n",
    "> if you think about it, finding the optimal solution would certainly mean overfitting!\n",
    "\n",
    "### Gaussian Processes\n",
    "\n",
    "Interestingly enough, Bayesian Hypermarameter Optimization relies on another machine learning algowitm (Gaussian Process) to **infer the latent accuracy (or loss)** distibution in the hyperparameters space, thus allowing a  more efficient search of hyperparameters which are looked after in the most **promising** regions. \n",
    "\n",
    "<img src=\"fig/escher.jpg\" width=\"400\"> <img src=\"fig/gaussian.png\">\n",
    "\n",
    "*Graphical representation of how to use a machine learning model to improve another machine learning model.*\n",
    "\n",
    "Gaussian processes tries to infer the **prior distribution** of a phenomenon (in our case, the accuracy function) by looking at some realisation of that phenomenon, sampled from its **posterior distribution**. In the situation shown above, the true prior function $f(x) = x \\sin(x)$ (red-dashed) is sampled at 6 different points (the red dots). Confidence bounds (the blue bands) for values in between the observation are set according to a *kernel* function. The default *kernel* in the **rBayesianOptimization** package is the **squared exponential kernel** $K_{SE}(x,x') = \\exp \\Big(-\\frac{\\|x-x'\\|^2}{2\\ell^2} \\Big)$, The parameter ℓ is the characteristic length-scale of the process (practically, \"how close\" two points $x$ and $x'$ have to be to influence each other significantly).\n",
    "\n",
    "___\n",
    "\n",
    "Bayesian Hyperparameter Search proceeds by fitting a *gaussian process* to a initial set of randomly sampled hyperparameters and then inspecting the most *interesting* areas in the hyperparameter space, where *interesting* can be defined by an **acquisition function** (e.g. *expected improvement*, *probability of improvement*, *upper confidence bound*), which according to the current best hyperparameter estimate, calculates a new set of hyperparameter which is expected to maximize the accuracy gain.\n",
    "\n",
    "Once a new hyperparameter set is choosen by the acquisition function, the underlying *gaussian process* is refit and the process iterates.\n",
    "___\n",
    "\n",
    "The function $\\texttt{BayesianOptimization}$ from the **rBayesianOptimization** packages, automatically takes care of the underlying gaussian process. It must be passed the following inputs:\n",
    "- $\\texttt{FUN}$: the target function to be optimized. Must take in input the hyperparameters and output a list consising of\n",
    "    1. Performance metric\n",
    "    2. Response\n",
    "- $\\texttt{bounds}$: a named list specifying the search space for each individual hyperparameter\n",
    "- $\\texttt{init}$_$\\texttt{points}$: the number of randomly sampled hyperparameters to initialize the gaussian process\n",
    "- $\\texttt{n}$_$\\texttt{iter}$: the number of iterations beyond the initially randomly sampled sample\n",
    "- $\\texttt{acq}$: the acquisition function of choice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 307,
   "metadata": {},
   "outputs": [],
   "source": [
    "willBeOptimized = function(K, KERNEL, DIST){\n",
    "  \n",
    "#  partition = titanic_df$x_train %>% nrow %>% seq_len %>% createDataPartition(p = 0.9, list= FALSE)\n",
    "  \n",
    "#  x_train = titanic_df$x_train %>% slice(partition)\n",
    "#  y_train = titanic_df$y_train %>% slice(partition)\n",
    "    \n",
    "#  x_val = titanic_df$x_train %>% slice(-partition)\n",
    "#  y_val = titanic_df$y_train %>% slice(-partition)\n",
    "    \n",
    "    df = get_titanic_df()\n",
    "    \n",
    "  kernel = c('rectangular', # 1\n",
    "             'triangular',  # 2\n",
    "             'epanechnikov',# 3\n",
    "             'biweight',    # 4\n",
    "             'triweight',   # 5\n",
    "             'cos',         # 6\n",
    "             'inv',         # 7\n",
    "             'gaussian',    # 8\n",
    "             'rank',        # 9\n",
    "             'optimal'      # 10                         \n",
    "  )\n",
    "  \n",
    "  kernel = kernel[KERNEL]\n",
    "  \n",
    "  model = train(x = df$x_train, #x_train,\n",
    "                y = df$y_train$class, #y_train$class,\n",
    "                method = 'kknn',\n",
    "               # trControl = trainControl(search = 'grid', method = 'none'),\n",
    "                ks = K,\n",
    "                tuneGrid = data.frame(kmax = 1,\n",
    "                                      kernel = kernel,\n",
    "                                      distance = DIST),\n",
    "                trControl = trainControl(method = 'cv',\n",
    "                                                   p = 0.9,\n",
    "                                                   number = 10)\n",
    "  )\n",
    "  \n",
    "\n",
    "acc =  model$results$Accuracy %>% max    \n",
    "test_response = model$finalModel$data$.outcome\n",
    "\n",
    "    \n",
    "#  test_response = model %>%\n",
    "#    predict(newdata = x_val) %>% factor\n",
    "#  \n",
    "#  acc = confusionMatrix(y_val$class,\n",
    "#                        test_response)[['overall']]['Accuracy']\n",
    "#\n",
    "#  test_response_probabilities = model %>%\n",
    "#    predict(newdata = x_val,\n",
    "#           type = 'prob') %>% pull('Yes')\n",
    "#     \n",
    "#  roc = roc(test_response_probabilities,\n",
    "#            y_val$class) %>% auc\n",
    "#  \n",
    "  return(list('Score' = acc,#roc,\n",
    "              'Pred' = test_response))  \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 329,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "elapsed = 3.60\tRound = 1\tK = 6.0000\tKERNEL = 10.0000\tDIST = 8.9064\tValue = 0.7625 \n",
      "elapsed = 4.25\tRound = 2\tK = 16.0000\tKERNEL = 5.0000\tDIST = 6.9588\tValue = 0.7690 \n",
      "elapsed = 4.37\tRound = 3\tK = 9.0000\tKERNEL = 7.0000\tDIST = 6.4410\tValue = 0.7690 \n",
      "elapsed = 6.22\tRound = 4\tK = 18.0000\tKERNEL = 6.0000\tDIST = 9.9433\tValue = 0.7886 \n",
      "elapsed = 6.55\tRound = 5\tK = 19.0000\tKERNEL = 2.0000\tDIST = 6.5915\tValue = 0.7853 \n",
      "elapsed = 5.06\tRound = 6\tK = 2.0000\tKERNEL = 9.0000\tDIST = 7.1145\tValue = 0.7330 \n",
      "elapsed = 5.07\tRound = 7\tK = 11.0000\tKERNEL = 3.0000\tDIST = 5.4863\tValue = 0.7766 \n",
      "elapsed = 5.99\tRound = 8\tK = 18.0000\tKERNEL = 1.0000\tDIST = 5.9820\tValue = 0.7864 \n",
      "elapsed = 4.95\tRound = 9\tK = 11.0000\tKERNEL = 4.0000\tDIST = 2.9627\tValue = 0.7722 \n",
      "elapsed = 5.24\tRound = 10\tK = 10.0000\tKERNEL = 10.0000\tDIST = 1.5564\tValue = 0.7864 \n",
      "elapsed = 4.86\tRound = 11\tK = 18.0000\tKERNEL = 4.0000\tDIST = 6.8924\tValue = 0.7820 \n",
      "elapsed = 6.83\tRound = 12\tK = 20.0000\tKERNEL = 10.0000\tDIST = 0.1000\tValue = 0.7798 \n",
      "elapsed = 4.17\tRound = 13\tK = 5.0000\tKERNEL = 1.0000\tDIST = 3.6322\tValue = 0.7723 \n",
      "elapsed = 4.33\tRound = 14\tK = 15.0000\tKERNEL = 10.0000\tDIST = 9.7642\tValue = 0.7897 \n",
      "elapsed = 6.73\tRound = 15\tK = 20.0000\tKERNEL = 6.0000\tDIST = 1.1837\tValue = 0.7973 \n",
      "elapsed = 2.50\tRound = 16\tK = 1.0000\tKERNEL = 1.0000\tDIST = 10.0000\tValue = 0.7265 \n",
      "elapsed = 6.93\tRound = 17\tK = 8.0000\tKERNEL = 1.0000\tDIST = 1.1639\tValue = 0.7973 \n",
      "elapsed = 4.52\tRound = 18\tK = 8.0000\tKERNEL = 3.0000\tDIST = 1.9263\tValue = 0.7755 \n",
      "elapsed = 5.73\tRound = 19\tK = 10.0000\tKERNEL = 1.0000\tDIST = 0.5429\tValue = 0.7907 \n",
      "elapsed = 4.83\tRound = 20\tK = 20.0000\tKERNEL = 7.0000\tDIST = 6.5138\tValue = 0.7723 \n",
      "elapsed = 5.55\tRound = 21\tK = 6.0000\tKERNEL = 10.0000\tDIST = 0.3581\tValue = 0.7788 \n",
      "elapsed = 5.16\tRound = 22\tK = 19.0000\tKERNEL = 8.0000\tDIST = 2.3520\tValue = 0.7842 \n",
      "elapsed = 6.90\tRound = 23\tK = 20.0000\tKERNEL = 7.0000\tDIST = 0.1000\tValue = 0.7602 \n",
      "elapsed = 4.69\tRound = 24\tK = 4.0000\tKERNEL = 6.0000\tDIST = 2.6157\tValue = 0.7418 \n",
      "elapsed = 4.92\tRound = 25\tK = 8.0000\tKERNEL = 2.0000\tDIST = 2.8654\tValue = 0.7668 \n",
      "elapsed = 5.77\tRound = 26\tK = 16.0000\tKERNEL = 3.0000\tDIST = 1.6284\tValue = 0.7973 \n",
      "elapsed = 5.60\tRound = 27\tK = 16.0000\tKERNEL = 9.0000\tDIST = 1.2518\tValue = 0.7962 \n",
      "elapsed = 4.75\tRound = 28\tK = 20.0000\tKERNEL = 1.0000\tDIST = 7.7618\tValue = 0.7820 \n",
      "elapsed = 5.57\tRound = 29\tK = 3.0000\tKERNEL = 1.0000\tDIST = 0.4863\tValue = 0.7723 \n",
      "elapsed = 5.20\tRound = 30\tK = 12.0000\tKERNEL = 5.0000\tDIST = 1.0046\tValue = 0.7820 \n",
      "elapsed = 5.12\tRound = 31\tK = 20.0000\tKERNEL = 3.0000\tDIST = 2.1605\tValue = 0.7930 \n",
      "elapsed = 5.65\tRound = 32\tK = 20.0000\tKERNEL = 1.0000\tDIST = 1.1527\tValue = 0.7820 \n",
      "elapsed = 4.75\tRound = 33\tK = 20.0000\tKERNEL = 8.0000\tDIST = 8.8203\tValue = 0.7831 \n",
      "elapsed = 5.42\tRound = 34\tK = 19.0000\tKERNEL = 1.0000\tDIST = 4.3337\tValue = 0.7788 \n",
      "elapsed = 2.69\tRound = 35\tK = 20.0000\tKERNEL = 10.0000\tDIST = 10.0000\tValue = 0.7919 \n",
      "elapsed = 4.50\tRound = 36\tK = 10.0000\tKERNEL = 10.0000\tDIST = 4.9256\tValue = 0.7755 \n",
      "elapsed = 5.42\tRound = 37\tK = 12.0000\tKERNEL = 1.0000\tDIST = 1.5678\tValue = 0.7853 \n",
      "elapsed = 2.67\tRound = 38\tK = 12.0000\tKERNEL = 3.0000\tDIST = 10.0000\tValue = 0.7723 \n",
      "elapsed = 5.81\tRound = 39\tK = 18.0000\tKERNEL = 6.0000\tDIST = 1.6977\tValue = 0.7929 \n",
      "elapsed = 2.61\tRound = 40\tK = 13.0000\tKERNEL = 1.0000\tDIST = 10.0000\tValue = 0.7821 \n",
      "\n",
      " Best Parameters Found: \n",
      "Round = 17\tK = 8.0000\tKERNEL = 1.0000\tDIST = 1.1639\tValue = 0.7973 \n"
     ]
    }
   ],
   "source": [
    "set.seed(123)\n",
    "\n",
    "#bayesian_opt = BayesianOptimization(willBeOptimized,\n",
    "#                          bounds = list(K = c(1L, 20L),\n",
    "#                                        KERNEL = c(1L, 10L),\n",
    "#                                        DIST = c(0.1, 10)),\n",
    "#                          init_points = 10,\n",
    "#                          n_iter = 30,\n",
    "#                          acq = 'ucb',\n",
    "#                          verbose = T)\n",
    "\n",
    "#saveRDS(bayesian_opt, 'data/bayesian_opt.R')\n",
    "bayesian_opt = readRDS('data/bayesian_opt.R')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 330,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Round</th><th scope=col>K</th><th scope=col>KERNEL</th><th scope=col>DIST</th><th scope=col>Value</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>17       </td><td>8        </td><td>1        </td><td>1.163896 </td><td>0.7973244</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllll}\n",
       " Round & K & KERNEL & DIST & Value\\\\\n",
       "\\hline\n",
       "\t 17        & 8         & 1         & 1.163896  & 0.7973244\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Round | K | KERNEL | DIST | Value | \n",
       "|---|\n",
       "| 17        | 8         | 1         | 1.163896  | 0.7973244 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Round K KERNEL DIST     Value    \n",
       "1 17    8 1      1.163896 0.7973244"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bayesian_opt$History %>% as_data_frame %>% top_n(1, Value)\n",
    "\n",
    "best_round = bayesian_opt$History %>% as_data_frame %>% top_n(1, Value) %>% pull('Round') %>% max"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see in the plot below, the best hyperparameters maximize the accuracy along the tree different dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 358,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAFoCAMAAABJ+DwrAAAAP1BMVEUAAAAaGhozZv9NTU1o\naGh8fHyMjIyampqnp6eysrK9vb3Hx8fKysrQ0NDW1tbZ2dnh4eHp6enr6+vw8PD////roB9Z\nAAAACXBIWXMAABJ0AAASdAHeZh94AAAdPUlEQVR4nO2diXbrKgxFeXGHdEja3uT/v/XFM4ME\nYvBEzlnr3iSAJRl2FYwdW90hqCKprQOAoJIC0FBVAtBQVQLQUFUC0FBVAtBQVQLQUFUC0FBV\nAtBQVQLQUFVaAOhG/U3v/1RjulP9P6NA1+2dKqWkBr1+ci1u70q9CAKGKtICQH+o8/T+rD5M\ndyGg+88xQD+QZlq8eeqgSrUA0HpW1rN15872RwMt0djyp1FMjlbqV2oMqkVLzKFf1WV4d7Ez\n5AJA33+4NCy3BVWjJcb8ot6Gd2892ueXx3f/T+dunnJcXlXzMXyYGnRTiKnJ73ujmvffYcNz\no14us5cZ1/7dz+vDxqX/fHtRb2qwZZlpq9otPnprn4+XPsEbQc6+HmGq99vd9ADtVosksUb1\nBNz6yUfTs9WiMNN67sreuw9zAwPoi9I3fJ3eD5GbQH/1bc/d5wfN5wlo00xb9Xj56Ip+3ruX\nTzvI2VdvptsNzQO0Wy0C9HmY1X52o//RHRh+dBODidZfpb4erDXtB7vB3OTjdr89kPprS5qf\n++1Nm11MQHfzmt8uu/6+9n8Vr7e5hWWmq3pYu7RhNv3Lix3D5Ou3bXh7bfdD9wDtVosA/Tus\nlr10R2UvasZrAnpg/tYot8HUpE+G7+1rny9v2rR4fPv4o+io7Gzc2tnOI/NqLSwzw6RisHYZ\n25kxTL7euzBv7f7oHqDdapnjppeOm59pFfj38vFqAj0A9Jhluw2mJv0KyV9rZsDXAHrUuWs8\nam7Uv9JmtBQ+zdjNGHTO75YHaLdaZnS+VHt+5L2dVdy7A68RhIm3EYtX5TawmuhHki7QzdtF\n+0QA7TGjvzgx2Gvmugdot1pmdG4DGF16+3xMXM9ffzTQ7avdQAq0sR/KfR8DtBuDC3RSV0Dr\naqFRapNzn6bH+cedBPrWf6+bDaRTDt1ho51EkUw57Bc3hu6lmaccDU7THEELAd1On1+MY7OL\nCfRbj8ensUh3N4A+D38Qw9GcZst+2zXqGv+2SxMm0B4zTiq+2EC/D8eHjekB2q2W+h5t1GU8\nA/7SrhT0C3QzO5+q+eqXdu0Gf0OzR0pV53697TcM9G93bPjb9Mt2eguPGSNDm0EOL4+d+J2W\n7WYP0G61FNDtiYvhuqTP8WDqR0+G/bmLj2H+OjV4YU+s9OGyc9qLfmLFaMGbMefQRpBjbX9i\n5dXyAO1WSwH9mBxP08/P9szzz6VfIp54+ZpPfWsNfl/axD40MU99ay/W205/5/EstgU0b8Za\n5dCDnGo/HwGdbQ/QboVDd6gqAWioKgFoqCoBaKgqAWioKgFoqCoBaKgqAWioKgFoqCoBaKgq\nAWioKgFoqCoBaKgqAWioKgFoqCoBaKgqAWioKgFoqCoBaKgqAWioKgFoqCodBej/enVvx89z\ncV9xaP1nvBxLc+zzcGgDQ1YvFcthgJ5fpj46MgOO/tP+P5rm2P+zygaMp3dOdXEdDOihVwyU\njwmBLf1P9Gj6z/zvrr/rRuu/6Z1dXVyHBtqsOrbM/HYszbEDaKksoPVZ81ExMLXkvHJp2d+Z\n+rs5AwFoXWb/dO/+czrxyPpvTGQH1Bw7d1A4vuCgcJQL9FR4VAxM6esBR9McuzU2xtEiMrSu\n+oGe/jue+INCvQZA63qOVY6D7gu/bAegOWkITyublc2h9ZdjaY7dIVZbAAHQupgzhdPnw+vI\nezPH7h71Gd+nzjFj+ViOAjQEiQSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqtBzQ\n/6KKI5uXsRLbPNtw/AbFDAV3iq9PqNnOGICOaJ5tGEAD6CWtAGhRPYD2hQGgtzEEoHMFoEts\nAKAB9HLNsw0DaAC9pBUALaoH0L4wAPQ2hgB0rgB0iQ0ANIBernm2YQANoJe0AqBF9QDaFwaA\nLlJ+/b4W8hyuB9C0s+tDALpI+aMnv699f2Z7DtcDaNLZtR8BAJ1ffp2AdpkG0EuJBvoKoLPL\nrwbQ1/AG/opgPYCmnI3d/y1rnlb8FEBfLaCvoQ0CFcF6AE05A9BJG5BHIhbQBtIAWqbmIfu9\n/epzdp0HQhxbfPGzAq0hDaBFaqb/5vf2q9cZgE7bgDkScYC+chuEQwrWA2jCGYBO24CZuLlA\nX+kNBCEF6wG068wYCGls8cXPDPQ10XO4/tmB/kfo2xDVYpfy7tMG+vZKaGRvO1VCWUCPB4HJ\nGZrM0cjQgvKr1Y9EjkaGFilzymEPhDxqAK3pavcjgTSAFqkw0PJrEAD0LANdGmj+0gIAbSgP\naGIgpFED6EkmuQzQ7KUFANpQMaDVMBCO32cCmuj0sCEL3BloZZZznj0jrbwhazW2jTgGxW6k\nFRlAz2cF9ffSM4Uaz0qNr7KoawTa3fkcoMcuDQBNOTWqBNg4NqIYFLsRV+QAHS8mQ6sxrYRj\niy8+BtBJGdqeWrAZmjlxdfgMTcW/C6Cn/g/GFl98EKATym2e+QzNnLg6+hya/IbZCGhjILTe\nD8UWX/yMQDsZ2skVgZCC9bsAek8ZmhkIAC0vd3hmVzm4ZVEp0MTm+wBaWLMl0Fe6OWNFUlwr\n0Cy4bHlESIynwJaDs8jdAdAA+k7yHAJavNBv1Fu2yC3JIOTOqgZa8EsLAE3ivA7QgxX68J4K\nQuCsAqDHvT210o7Ohw5QbGzxxTUCzYB76ruTBnrsW12hkWb/dNjjT/4KKXp3dnViJV420P0I\nnMY+6Feb5uUYAE2X8zBrshgb+1aX58TKGAHDM7tCyH9NkCfgy55Y4dysBrTT+Wr4c95hhpZ3\nyuJAE8g4OBtITxnaRjqcoYNAizL0XGF7KJOhdTfEPq0J9DAYekRc1InFZZoHk1lyHNEbEMjo\nCBNMc8coJTK0vIY9WswD2vJC7dNaQOtDQHb5joDeS4amkJnY/XYK+441N9CsBTN0PLYCoO0o\nEoFm3GySofsvqolji2inOWNFVnzcObQ7DiQycyr+dsu7Ku72Br6d6ucCHJz5QOuxRALt97/J\nHLo7lNC+EpkU/dxAm1+eLBnc1GKqcxc+BDvVO0/ANhLooUbQNVJj2wDdZui5o7+ZFL0ToAMH\nLplxTDNVu9g53CIH0zmmNuUnOpShVwO6oLGNphzGSHwzk459AB1aWsqMQ8NmKvEPmj1VnobS\nXkGbWhCL0/6Q7Mi8MQhr1jG2yUFh51nr5W/jUyujOWNFWLzzDO0ZHEG5ySqX0lmkAzu1VlI9\nfIbuHJtAk0TvBGhv8zzD3sEJlxMnUMgNmCzN/9hwiO6IQG8wh+78nlygHaKrB9o/OMFyu8/4\nDRii/Tu1GoMHB7r365zJIojeCdBcP8X54wznAS3egEQaQKdquj/TdI+qR+9ad62ySja8iZQm\nNxx3n7JtJ+rk9mGoubWBZ6dyg9tM7j4tmqHHvyP3WgMiRW+doQN/+HH+7PJwtgmUuxlXsIG5\niWenvAaRoQcZQDvR2QO0JdCSforzZ5ULBifMZ9wG7nIHgM6UF2jlpOiNgJb2U5w/s1wyOFS5\nunJfaDJD5la+Eys+gwB6kA9opZRzkMNZiSmOas531T6A7s6dkMfQUkNGkvae+vYYBNCDJqCp\n9VNFLEMxVmKKpc0DXVUWaNHg+DI0uQonMqTPpJGhM+UF2iruUxFthTEeVc7xtQbQssHxlNPn\nSWSGTvPc7kBzaOnvYnYH9Fjefue15aQVxnhUuX7WUtBVJYGWeQycJ4naQFc/tzsFdsprcH2g\n7etUDgK0Xj5OFnPvsss3Z/qQviKiHND8qInLaZ6lhtSUOTwj7TWIDD1IArS7nNdLt8IYjyjn\nu4S+Zm1XQDMJOsZQN5H2/QTLaxAZetAItJmIlRYdD3SnwQpjPFyurIf6EewyGbrYT7AIj8zQ\ncAeFHc9UlF5D1gY90fxe5B0UUtFx1wLWkKGNqbLxx0ecQST2kjHOOe00eDOMU9mYdlrsR7Lk\n/nD7SYUx8Ex9j/gMORv4M3Tesh0VHfPNFzYWucnOMrR70ZJ/ZwzjjpzmyrIip6JUhg7sT6hc\njYtu3gxte6Z3tWCGPllSdgFTtoA2BrorzgA6XFyoeaE5dGYk/Zgx7bmvLvZPiRr8Ud7IzJp1\nQJVqD0D/s3rHM6LxxbsCOiOSecjo9gHPd8p7PtAbUevRykCrsRumUbhbK2inioFmPY6jYZTT\nA0ZEKPA8yNowc9nOiCrQf6stmaz7E6z+yMDi2RxqrXtqA5rzGJF/qAgFnjVpG+Yt21E0+7ZZ\nB+i1fyTbZWjlAm0S7Y9aK1Z0sWTf1weajFr+pc1FKPBsfpzd5xwUUjB7t6kyQ4+LZ30/zMV3\njei5m4IoGktBhwFaj9qklUPYE6HIs/FJc59xtR0b4JNl6N7pyQf0VQ70oTI0cZLAT64kQpFn\nO0NPfZ18tV1C1BVn6JFYrbjVFNLUWVXNoQnDA82RkWjlMs/sCn3yKof4+0NUcfwTKwGgJ6Ir\nB3rEYguggz9l90TgnQ8dGWjt8cdN038YX+8N+Wzkf+NNwNoXp8PnoPwjvSrQ46wgF2jXYeiL\nKFwu88z/OMcHNDsAfXcUZXAnQDsPqG+0V+JB33c50MNgbwU0tcaQCbTjUMtyqUDLPLPlVx/Q\nbEYJTJNqArphyjXpQBvFYw9fxyMmX6/RxSoXaG65rATQrkPtW/t73m1zjwI7JPLMl18DQJMZ\nxZ0mxT2SolTNEgeFPqBpngNAGxfExQLNX8wVsOLneKIuC2g3Dn0W+k2Hz+7RYEjk2VcemHJQ\nQLvTJPlDg3a/bGcBbSZoawpt3JCnnYKRN0bq7nyj+hvgdBO1GKnYW+x4IBbdkCfnFkmuEyJ8\n/x7J3fNxeXaK6gWyb6L7fWEpZ5+KAG3WTd3V5qr+r9wsHqT/qek5TPTXKZ5y+HPxAqscjmFz\n39Lm0CLP/nK+gszQ+hfWE8yhG+vVB/RVADRPdCLQgTlFwEwG0I5hy2sS0CLPgfI4oI3Oqh9o\nl+cEoF2iI37czI6/kGS/mcJAiwL3lMs8B8oDQJ8coMOx1Qq0Z8pxEgPNES0G2kvyaidWHMP2\nTiUALfMcKvcB7RyVy6ZJNQNNrHQMQLdrPUy/mlFTRCsR0DbDzm3GYjGK/AnW3FwLfA4tOZLh\nkiGP54hy30grC2h2mlRy2c6JalWgpzOFBtmNWWdqBNpaW2GBJpL0Y1Mv0HRGJpbA4oCO/JHs\n3Fy3MAWYHMlwbzuP55hy/49kT/r5QDvqost2ztWAVORpxnqL3F5m69996BllFdNxf18nQOdC\nPkP75hbSX3eXztC6hf7F/c6JztBezxHl3gytM+x+U5bJ0HR8WlTcilEW0C8fP/x+x6kHWtsV\nN2ynC9xDONE6XLCHV5lDu4aJSVT0HFriWVLunUPPf3p2h/piEwMtC4OtyQH68RXQvF/4XY+Q\nAGjiTuA2o+YXIU+yv4fXANo1TEUYC3Q0n4lAD2NFdms60FFhhCKX9Y4F9O3rrb0ZzOvXH7/7\nQv27j33Dh032DUWugGVfD28GdG4k8XxmAB2ZJXw1ogcgp0RuupHNoS/n5sH0S26e/kfyLAA6\ngLS3J7cD2jUcBQdV7nQX24+C8hDQ7O8GE4BOwjYq8sGR8KDw79zds+uVdS3Sv7v14ysqOq5v\nQjl5d0AThkk67O9is9zjQFqRDHTCLwfZP8PFgR7CFgH9+9al559X9cb6lmj6Jb8vumCvlSF0\nC6BpPLgzf8X4TAea6SNxjcxZQaDJGhvoy+s025CvxjLOiB2tFWjXMPv17TckLd8T0FHOVgb6\nRam337GKudBZKBHQ1h1vxZ25M6Bdw9zXdyyHOwY6zdnKQKvzr9smTU8EtGuB5tndfa5bguXb\nAZ3pbGWgb+y20ZIBbV/OIzwXtSugCQscz/bFH2y3hMqXB9q8u7a2C3nO1p5Dv3UF6qXEOjTZ\nCX6gpVcL7AlowgCZoKcNqMsqdge0NhLfxh5kOlsZ6HPf10q9s0akEgJtnbA/YIYmDBA867t/\nzAxdxtnKQDequ5bjN3OFo3dG9oMbQwqK0UBzMRoxjIoCmnBHJGh+99PKiwJN/FDd7dRCztY+\nKFTma4bEQHuPSbKA9jn1lccATUXh8Bw2vXegizlbGeg39f44Lrydc08T3mOA9vVnGtBuLFyM\nMcVUS4ZnAF2wJgfov6Z/VFWTv3oXAbR7r6F0oJlYuBijmrsi4zjZd89Kj2QvQJdztvYqx+38\notTLOX+RYwOgOevLAU3GcbKBzohkJ0BH29wP0OUUAzRxe8NIoL3WlwKaDu9k/4Q6IxIAHbvJ\nToCmc50MaIH1ZYCmwzvZN7nIiWQfQLPbHQDo/srRiJ+JEl7Mu2KJ71WVpux7ZPnk7lMw4m7C\nsVqAKSJ2asMuLiMW6InnYst2brG3eVyGJqwwxqPKZRmaDG9Y4WDPFu8zQ3sOYq6+Zfn9Z+hG\n/b6qv9uryv+tbDTQJNE80JyViOIsoOnwTvbzCPIi2QPQizNY1Jh7YuVDXe63cuvQ0ujacjnQ\nhQgtDvTJBjozkh0AvTyDRY25QF/UZ6EzhSp6RIVAFyM0A2g6PPs3j7mRFATaO6I80D6TBRj0\n3pcj1phlsNWb+vpTL/efEkD312tJo+vLJUCHrUiLiwPt/Ig3N5JyQPsPi7YCmrlzUpqx3qL5\nsSX5tT0mLHG1XXyGvpO/NjVZFlmJdCpsron+e5vOebMXqB0tQ3tNHiBD3y8v9/u7UmfWhlj9\nHFoSg1nOAx1jJdKprLkmEujTnoFOm0P7t9z/HLqgUoGm7qnlGNoaaOrvzX1yeX4kWwMd2HL/\nQL/mTzVmZ6lA2/efCbQWlK8D9AlAh52tDHRTLmNnAN1q6NFlCU0F2v0CMRN0X14gkrWAZi5L\nCW25f6B/X0tcaDc4ywI6qXgloM0vEIJnAB2/yTJAq9KnviUx+MqPA/TJBrpEJNsCHdzy+YAW\nxeArPwzQ5s9UAHTBmj2tcohi8JUfCeirUV4kkk2BDm8JoKsF+nRwoKkL0sNb7h9oTDl8zQeZ\n4FIJmv+Rx4GAFmwJoJ8GaG7b4wAt2XL/QPf6e/1gbYjVAi2LwVd+EKCdG7Jz2wLohY3Rmfim\n8ol+NqANGopxCKDLHBQWmnLIYvCV7xBoIxPTCfqAQNu3GBFteRigvzJvdt47K3Gq7DBAmzQA\n6II1RQ4K868ffSKgnQR9fKBlWx4G6MbhWXued9MMH/RXcRiVAm3RAKAL1pSYQ9synlk/vm+Y\nOm8YFQJN3JsRQBesWQNo/fOTAW2M+zc54Tgk0P+sXZBseQCgmUdSUEA3TJ0/jOqAJm+eC6AL\n1uQAzT2SwoK2/zhOoc06+rZZxxS1T8S9v4515yxuoA60C45YoLlHUpBAD/89d4am725+8Awt\n3XL/GZp7JAUF9PjuuYA24f1mHj9xRKDvDM+HBpp7JIUJrTmZfmqgqSNCnwUAvbAx4SMpKKCf\nc8rhAk0k6GMDLd9y/0Bzj6TggKYOCv1h1AU093ygIwMdseUBgOY0ng00FqDHsmc6U+gATfEM\noOM3WRnoeAHoApGsCjR9/+ODA13yWd8xxccEmjkkPCrQ9G+MDg100Wd9xxQfEmj+CZsHBbpY\nzW6ALvqs75jiIwKd8EhvAL0y0EWf9R1TfECgnQdsSn4oDaAXNrbks75jio8JtP1DvLAFAL2w\nsSWf9R1TfDygTy7QAgsAemFjSz7rO6b4kECzT8wA0JsZwzp0RPNW5hEh90xvAL2VMQAd0byV\nkaAB9O6M0UD/nEvcxiCm+FFOhrJXoN1HILc8q6CFZwZalTTG1RAUXd4fR4brA03fT2/HQF8d\noOddANCu+t5ZG+jLe/eUwgtrQ6yqM7TzCOR+woEM7atYP0P3NCt1Yy1EqOo5tP3E2Kv0d6XP\nDPQqxnSgh9xc4CyhJ4wqgHYegQyg92LMBPrtdi9y2tsTRg1AO0+MtXkG0JsZQ4aOaN5Kn3AA\n6P0Zwxw6onkrI0ED6N0ZY1Y5flgbYlUN9NUGWmoBQC9sbD/r0BtYAdCi+qMDfc88U7j1baEK\nitin7t5f1s2/jnXzrKoHCtdyxDRvRTzT203QyNCbGQPQEc1bAeg8ZwB6QSsAWlQPoH1hHB7o\nEwG03AKAXtgYgI5o3sq8twyA3psxAB3RvBUFdIQFAL2wMQAd0bwVgM5zBqAXtFII6BgLAHph\nYwA6onkrF+goCwB6YWMAOqJ5KwCd5wxAL2glBWjzfnYAem/GAHRE81YO0HEWAPTCxgB0RPNW\nADrPGYBe0AqAFtUDaF8YtQEdaQFAL2wMQEc0bwWg85wB6AWtAGhRPYD2hVEZ0HGGATSAXtJK\nAtDWY1UA9N6MAeiI5ncAne0MQC9oJR/oxTkE0AB6ueZ3AJ3tbC9Aa8/zbprhg/6s76oeXu/r\nFKXzvDyHqwCdep/bAwOtP7N+LJjK6Ht4HBdo+tbrU111QCffibwioJt7xUCLM/QKHCJDrwf0\n+Ia5ydKBgfZNNw2eKwE6UP8EQBvv7Cn01reFKihin455+y9dVQ9ULtCNWzd1F22pngydYBgZ\neu8Z2n0TCgNAFykH0ExNEtAExs8HdIphAL1voBuiLhQGgC5SDqCZmhygtRdipQNAF4gEQMdu\nEnumUCPbOWMoCqMWoJMMA+jdAB0vAF0gEgAduwmAjmjeyuAZQO/OGICOaN4KQOc5A9ALWkkH\nOtEwgAbQS1pJBjrVMIAG0EtaSQU62TCABtBLWgHQonoA7QsDQBcpB9BMDYCOaN4KQOc5A9AL\nWkkEOt0wgAbQS1oB0KJ6AO0LA0AXKQfQTA2AjmjeyrwhNIDemzEAHdG8FYDOcwagF7QCoEX1\nANoXBoAuUg6gmRoAHdG8FYDOcwagF7QCoEX1ANoXBoAuUg6gmRoAHdG8FYDOc3ZAoLe+LVRB\nEft02DuATap6oJChY5q3QobOc3bADO0PA0AXKQfQTA2AjmjeCkDnOQPQC1oB0KJ6AO0L4/BA\n5xgG0AB6SSsAWlQPoH1hAOgi5QCaqQHQEc1bAeg8ZwB6QSsAWlQPoH1hAOgi5QCaqQHQEc1b\nAeg8ZwB6QSsAWlQPoH1hAOgi5QCaqQHQEc1bAeg8ZwB6QSsAWlQPoH1hAOgi5QCaqQHQEc1b\nAeg8ZwB6QSspQGcZBtAAekkrAFpUD6B9YQDobQwB6FwB6BIbAGgAvVzzbMMAejdAa8/zdp7x\n/VTP+s4yDKD3ArT2zPqpoLFeZWEA6G0MAWhDNrTNHUAnGAbQAHpJKwBaVP8EQDf6fzbQW98W\nqqBq3Kcqd2pRoMfuoi0hQ29jCBnaUALQELSBkoBujDIADe1HABqqSgAaqkqxZwp1ev1nCiFo\nAy13LQcEbSAADVUlAA1VJQANVSUADVWl8kDrax6C9Q+7ub99Y12XHbZtXsbtbU3G4/cQjlLS\nPNbZvIFoi9TVqLihcZoJ3cUNUteIjG54WxxoY11Pey9oLljMti74kyx/S40Ty5JCD6xLYWvH\nsWgDsZfU8wWRQ2M3i3IX48Y/UNUD3ThvuHYbAU04Fm1QE9DSQeqb7Brou9YkiudYoIUtSwAd\nu0lshk4JLAHo2VUKz0lAC7+hbPO7BTpuCi0ybkyzpF8WuUBHTlYTgY7ykgt0whRa7i5ikEzL\nOwVa3gXxxs0vM6HxEhk66bs9OkPHTDtSdieSzsTeixgks9XOgXY/+JpHAh02XgzosKuAY7nt\ntYCOcZYBtMzN3oGO3J0o4433I1FZJ9DxKZ0yvxTQMYNkWt4l0NbolJ1ypBmvbcrRJIUV13tO\nsxSga5hyNOZbf2u9RQLQYePFgI7ZLB3oqG/06N2JGhqnWSrQMX82awBtXDkdcTJP3lxu3Nhj\n2SpHtAdflNLmsRvFbeDc6UrqJW5ojJii3MUM0tyecYVrOaCqBKChqgSgoaoEoKGqBKChqgSg\noaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGq\nBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSgoaoEoKGqBKChqgSg\noar0P20E+oe41QvaAAAAAElFTkSuQmCC",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = bayesian_opt$History %>%\n",
    "gather(hp, Hyperparameters, -one_of('Value', 'Round')) %>%\n",
    "filter(Round > 10) %>%\n",
    "rename('Accuracy' = Value) %>%\n",
    "ggplot(aes(x = Hyperparameters, y = Accuracy)) +\n",
    "stat_smooth(method = 'loess', formula = y ~ exp(-abs(x)/x^2), se = T, level = 0.95) +\n",
    "geom_point(size = 0.5) +\n",
    "facet_grid(~ hp, scales = 'free') +\n",
    "theme_minimal() + ggtitle('Validation Performance') + labs(x = \"\", y = \"Accuracy\")\n",
    "\n",
    "options(repr.plot.width=6, repr.plot.height=3)\n",
    "\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 335,
   "metadata": {},
   "outputs": [],
   "source": [
    "set.seed(123)\n",
    "\n",
    "kernel = c('rectangular','triangular', 'epanechnikov', 'biweight', 'triweight', 'cos', 'inv', 'gaussian', 'rank', 'optimal')\n",
    "\n",
    "model_bayesian_opt = train(x = titanic_df$x_train,\n",
    "                            y = titanic_df$y_train$class,\n",
    "                            method = 'kknn',\n",
    "                            trControl = trainControl(search = 'grid', method = 'none'),\n",
    "                            tuneGrid = data.frame(kmax = bayesian_opt$History$K[best_round],\n",
    "                                              kernel = kernel[bayesian_opt$History$KERNEL[best_round]],\n",
    "                                              distance = bayesian_opt$History$DIST[best_round])\n",
    "                          )\n",
    "\n",
    "saveRDS(model_bayesian_opt, 'data/model_bayesian_opt.R')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Recap | Training Sample\n",
    "\n",
    "Let's summarise the in-sample performances of the trained models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 336,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Strategy</th><th scope=col>Accuracy</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Grid Search    </td><td>0.803          </td></tr>\n",
       "\t<tr><td>Random Search  </td><td>0.802          </td></tr>\n",
       "\t<tr><td>Bayesian Search</td><td>0.797          </td></tr>\n",
       "\t<tr><td>Manual Search  </td><td>0.782          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Strategy & Accuracy\\\\\n",
       "\\hline\n",
       "\t Grid Search     & 0.803          \\\\\n",
       "\t Random Search   & 0.802          \\\\\n",
       "\t Bayesian Search & 0.797          \\\\\n",
       "\t Manual Search   & 0.782          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Strategy | Accuracy | \n",
       "|---|---|---|---|\n",
       "| Grid Search     | 0.803           | \n",
       "| Random Search   | 0.802           | \n",
       "| Bayesian Search | 0.797           | \n",
       "| Manual Search   | 0.782           | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Strategy        Accuracy\n",
       "1 Grid Search     0.803   \n",
       "2 Random Search   0.802   \n",
       "3 Bayesian Search 0.797   \n",
       "4 Manual Search   0.782   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = readRDS('data/titanic_model.R')\n",
    "model_grid_search = readRDS('data/model_grid_search.R')\n",
    "model_random_search = readRDS('data/model_random_search.R')\n",
    "model_bayesian_opt = readRDS('data/model_bayesian_opt.R')\n",
    "\n",
    "data_frame(\n",
    "'Manual Search' = model$results$Accuracy %>% round(3),\n",
    "'Grid Search' = model_grid_search$results$Accuracy %>% round(3) %>% max,\n",
    "'Random Search' = model_random_search$results$Accuracy %>% max %>% round(3) %>% max,\n",
    "'Bayesian Search' = bayesian_opt$History$Value %>% max %>% round(3)\n",
    "    ) %>% gather(key = 'Strategy', value = 'Accuracy') %>% arrange(desc(Accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The main reason for which bayesian search has a significantly higher accuracy is because it can exploit non-integer value for the $\\texttt{distance}$ parameter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performance Recap | Testing Sample\n",
    "\n",
    "Let's test our models on the **hold-out** sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 337,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_response = model %>%  predict(newdata = titanic_df$x_test, type = 'raw') %>% factor\n",
    "test_response_grid = model_grid_search %>% predict(newdata = titanic_df$x_test, type = 'raw') %>% factor\n",
    "test_response_random = model_random_search %>% predict(newdata = titanic_df$x_test, type = 'raw') %>% factor\n",
    "test_response_bayes = model_bayesian_opt %>% predict(newdata = titanic_df$x_test, type = 'raw') %>% factor\n",
    "\n",
    "test_response_proba = model %>% predict(newdata = titanic_df$x_test, type = 'prob') %>% pull('Yes')\n",
    "test_response_grid_proba = model_grid_search %>% predict(newdata = titanic_df$x_test, type = 'prob') %>% pull('Yes')\n",
    "test_response_random_proba = model_random_search %>% predict(newdata = titanic_df$x_test, type = 'prob') %>% pull('Yes')\n",
    "test_response_bayes_proba = model_bayesian_opt %>% predict(newdata = titanic_df$x_test, type = 'prob') %>% pull('Yes')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 338,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Strategy</th><th scope=col>Accuracy</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Grid Search    </td><td>0.798          </td></tr>\n",
       "\t<tr><td>Manual Search  </td><td>0.796          </td></tr>\n",
       "\t<tr><td>Bayesian Search</td><td>0.788          </td></tr>\n",
       "\t<tr><td>Random Search  </td><td>0.758          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Strategy & Accuracy\\\\\n",
       "\\hline\n",
       "\t Grid Search     & 0.798          \\\\\n",
       "\t Manual Search   & 0.796          \\\\\n",
       "\t Bayesian Search & 0.788          \\\\\n",
       "\t Random Search   & 0.758          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Strategy | Accuracy | \n",
       "|---|---|---|---|\n",
       "| Grid Search     | 0.798           | \n",
       "| Manual Search   | 0.796           | \n",
       "| Bayesian Search | 0.788           | \n",
       "| Random Search   | 0.758           | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Strategy        Accuracy\n",
       "1 Grid Search     0.798   \n",
       "2 Manual Search   0.796   \n",
       "3 Bayesian Search 0.788   \n",
       "4 Random Search   0.758   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_frame(\n",
    "'Manual Search' = (acc = confusionMatrix(test_response, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Accuracy']  %>% round(3) %>% unlist,\n",
    "'Grid Search' = (acc_grid = confusionMatrix(test_response_grid, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Accuracy'] %>% round(3) %>% list('Grid Search ' = .) %>% unlist,\n",
    "'Random Search' = (acc_random = confusionMatrix(test_response_random, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Accuracy'] %>% round(3) %>% list('Random Search ' = .) %>% unlist,\n",
    "'Bayesian Search' = (acc_bayes = confusionMatrix(test_response_bayes, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Accuracy'] %>% round(3) %>% list('Bayesian Search ' = .) %>% unlist\n",
    ") %>% gather(key = 'Strategy', value = 'Accuracy') %>% arrange(desc(Accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Strategy</th><th scope=col>Balanced Accuracy</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Manual Search  </td><td>0.776          </td></tr>\n",
       "\t<tr><td>Grid Search    </td><td>0.775          </td></tr>\n",
       "\t<tr><td>Bayesian Search</td><td>0.763          </td></tr>\n",
       "\t<tr><td>Random Search  </td><td>0.750          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Strategy & Balanced Accuracy\\\\\n",
       "\\hline\n",
       "\t Manual Search   & 0.776          \\\\\n",
       "\t Grid Search     & 0.775          \\\\\n",
       "\t Bayesian Search & 0.763          \\\\\n",
       "\t Random Search   & 0.750          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Strategy | Balanced Accuracy | \n",
       "|---|---|---|---|\n",
       "| Manual Search   | 0.776           | \n",
       "| Grid Search     | 0.775           | \n",
       "| Bayesian Search | 0.763           | \n",
       "| Random Search   | 0.750           | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Strategy        Balanced Accuracy\n",
       "1 Manual Search   0.776            \n",
       "2 Grid Search     0.775            \n",
       "3 Bayesian Search 0.763            \n",
       "4 Random Search   0.750            "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_frame(\n",
    "'Manual Search' = (acc = confusionMatrix(test_response, titanic_df$y_test$class, positive = 'Yes'))[['byClass']]['Balanced Accuracy']  %>% round(3) %>% unlist,\n",
    "'Grid Search' = (acc_grid = confusionMatrix(test_response_grid, titanic_df$y_test$class, positive = 'Yes'))[['byClass']]['Balanced Accuracy'] %>% round(3) %>% list('Grid Search ' = .) %>% unlist,\n",
    "'Random Search' = (acc_random = confusionMatrix(test_response_random, titanic_df$y_test$class, positive = 'Yes'))[['byClass']]['Balanced Accuracy'] %>% round(3) %>% list('Random Search ' = .) %>% unlist,\n",
    "'Bayesian Search' = (acc_bayes = confusionMatrix(test_response_bayes, titanic_df$y_test$class, positive = 'Yes'))[['byClass']]['Balanced Accuracy'] %>% round(3) %>% list('Bayesian Search ' = .) %>% unlist\n",
    ") %>% gather(key = 'Strategy', value = 'Balanced Accuracy') %>% arrange(desc(`Balanced Accuracy`))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Strategy</th><th scope=col>Kappa</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Grid Search    </td><td>0.548          </td></tr>\n",
       "\t<tr><td>Manual Search  </td><td>0.547          </td></tr>\n",
       "\t<tr><td>Bayesian Search</td><td>0.525          </td></tr>\n",
       "\t<tr><td>Random Search  </td><td>0.478          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Strategy & Kappa\\\\\n",
       "\\hline\n",
       "\t Grid Search     & 0.548          \\\\\n",
       "\t Manual Search   & 0.547          \\\\\n",
       "\t Bayesian Search & 0.525          \\\\\n",
       "\t Random Search   & 0.478          \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Strategy | Kappa | \n",
       "|---|---|---|---|\n",
       "| Grid Search     | 0.548           | \n",
       "| Manual Search   | 0.547           | \n",
       "| Bayesian Search | 0.525           | \n",
       "| Random Search   | 0.478           | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Strategy        Kappa\n",
       "1 Grid Search     0.548\n",
       "2 Manual Search   0.547\n",
       "3 Bayesian Search 0.525\n",
       "4 Random Search   0.478"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_frame(\n",
    "'Manual Search' = (acc = confusionMatrix(test_response, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Kappa']  %>% round(3) %>% unlist,\n",
    "'Grid Search' = (acc_grid = confusionMatrix(test_response_grid, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Kappa'] %>% round(3) %>% list('Grid Search ' = .) %>% unlist,\n",
    "'Random Search' = (acc_random = confusionMatrix(test_response_random, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Kappa'] %>% round(3) %>% list('Random Search ' = .) %>% unlist,\n",
    "'Bayesian Search' = (acc_bayes = confusionMatrix(test_response_bayes, titanic_df$y_test$class, positive = 'Yes'))[['overall']]['Kappa'] %>% round(3) %>% list('Bayesian Search ' = .) %>% unlist\n",
    ") %>% gather(key = 'Strategy', value = 'Kappa') %>% arrange(desc(Kappa))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>Strategy</th><th scope=col>AUC</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>Manual Search  </td><td>0.8456319      </td></tr>\n",
       "\t<tr><td>Grid Search    </td><td>0.8293264      </td></tr>\n",
       "\t<tr><td>Bayesian Search</td><td>0.8253195      </td></tr>\n",
       "\t<tr><td>Random Search  </td><td>0.8123483      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       " Strategy & AUC\\\\\n",
       "\\hline\n",
       "\t Manual Search   & 0.8456319      \\\\\n",
       "\t Grid Search     & 0.8293264      \\\\\n",
       "\t Bayesian Search & 0.8253195      \\\\\n",
       "\t Random Search   & 0.8123483      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "Strategy | AUC | \n",
       "|---|---|---|---|\n",
       "| Manual Search   | 0.8456319       | \n",
       "| Grid Search     | 0.8293264       | \n",
       "| Bayesian Search | 0.8253195       | \n",
       "| Random Search   | 0.8123483       | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  Strategy        AUC      \n",
       "1 Manual Search   0.8456319\n",
       "2 Grid Search     0.8293264\n",
       "3 Bayesian Search 0.8253195\n",
       "4 Random Search   0.8123483"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_frame(\n",
    "'Manual Search' = roc(test_response_proba, titanic_df$y_test$class) %>% auc,\n",
    "'Grid Search' = roc(test_response_grid_proba, titanic_df$y_test$class) %>% auc,\n",
    "'Random Search' = roc(test_response_random_proba, titanic_df$y_test$class) %>% auc,\n",
    "'Bayesian Search' = roc(test_response_bayes_proba, titanic_df$y_test$class) %>% auc\n",
    ") %>% gather(key = 'Strategy', value = 'AUC') %>% arrange(desc(AUC))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Honest Toughts / Takeouts\n",
    "\n",
    "- A better understanding of the problem and of the relationship and a good definition of the model/models to start with does the best of the work\n",
    "    \n",
    "- The larger the problem, the more incentive you have in going towards \"smart\" approaches, cf. Random Search and Bayesian Optimization. It is unlikely you will ever face a problem of a truly large size.\n",
    "\n",
    "> The two above are especially true for complex problem, where the deadly combination of hyperparameter space size and training time are often called the **curse of dimensionality**\n",
    "\n",
    "\n",
    "- If the problem is simple and/or well known, you'd be better off with a quick and dirty manual search.\n",
    "\n",
    "- There is a lot of emphasis on gaining that little extra bit of performance (Kaggle competitions...), while a real-world task is pretty much aimed at finding a non useless solution. Basis-point improvement of AR are effectively worth nuts. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}