1_2_mlpackages.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "source(\"src/lib.R\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CARET\n",
    "\n",
    "![gather](fig/caret.jpg \"title-1\")\n",
    "\n",
    "**CARET** is a package that we are going to use extensively in the next lessons. It provides a unified framework to build predictive models leveraging on the large list of packages belonging to the **R** ecosystem. \n",
    "\n",
    "Each one of these packages has its particular interface and to use it you have to adapt your code to this interface.  \n",
    "\n",
    "**CARET** provides a standard interface with these packages: if you are using **CARET** and you decide to change the statistical methods / implementation in your code, you will simply need to change a few parameters and your code will work seamlessy.\n",
    "\n",
    "To use this package you have to simply load the package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(caret)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then retrieve all the models supported by **CARET** using the $\\texttt{getModelInfo}()$ function, "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Number of models: 238'"
      ],
      "text/latex": [
       "'Number of models: 238'"
      ],
      "text/markdown": [
       "'Number of models: 238'"
      ],
      "text/plain": [
       "[1] \"Number of models: 238\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'Supported Models: ada, AdaBag, AdaBoost.M1, adaboost, amdai, ANFIS, avNNet, awnb, awtan, bag, bagEarth, bagEarthGCV, bagFDA, bagFDAGCV, bam, bartMachine, bayesglm, binda, blackboost, blasso, blassoAveraged, bridge, brnn, BstLm, bstSm, bstTree, C5.0, C5.0Cost, C5.0Rules, C5.0Tree, cforest, chaid, CSimca, ctree, ctree2, cubist, dda, deepboost, DENFIS, dnn, dwdLinear, dwdPoly, dwdRadial, earth, elm, enet, evtree, extraTrees, fda, FH.GBML, FIR.DM, foba, FRBCS.CHI, FRBCS.W, FS.HGD, gam, gamboost, gamLoess, gamSpline, gaussprLinear, gaussprPoly, gaussprRadial, gbm_h2o, gbm, gcvEarth, GFS.FR.MOGUL, GFS.LT.RS, GFS.THRIFT, glm.nb, glm, glmboost, glmnet_h2o, glmnet, glmStepAIC, gpls, hda, hdda, hdrda, HYFIS, icr, J48, JRip, kernelpls, kknn, knn, krlsPoly, krlsRadial, lars, lars2, lasso, lda, lda2, leapBackward, leapForward, leapSeq, Linda, lm, lmStepAIC, LMT, loclda, logicBag, LogitBoost, logreg, lssvmLinear, lssvmPoly, lssvmRadial, lvq, M5, M5Rules, manb, mda, Mlda, mlp, mlpKerasDecay, mlpKerasDecayCost, mlpKerasDropout, mlpKerasDropoutCost, mlpML, mlpSGD, mlpWeightDecay, mlpWeightDecayML, monmlp, msaenet, multinom, mxnet, mxnetAdam, naive_bayes, nb, nbDiscrete, nbSearch, neuralnet, nnet, nnls, nodeHarvest, null, OneR, ordinalNet, ORFlog, ORFpls, ORFridge, ORFsvm, ownn, pam, parRF, PART, partDSA, pcaNNet, pcr, pda, pda2, penalized, PenalizedLDA, plr, pls, plsRglm, polr, ppr, PRIM, protoclass, pythonKnnReg, qda, QdaCov, qrf, qrnn, randomGLM, ranger, rbf, rbfDDA, Rborist, rda, regLogistic, relaxo, rf, rFerns, RFlda, rfRules, ridge, rlda, rlm, rmda, rocc, rotationForest, rotationForestCp, rpart, rpart1SE, rpart2, rpartCost, rpartScore, rqlasso, rqnc, RRF, RRFglobal, rrlda, RSimca, rvmLinear, rvmPoly, rvmRadial, SBC, sda, sdwd, simpls, SLAVE, slda, smda, snn, sparseLDA, spikeslab, spls, stepLDA, stepQDA, superpc, svmBoundrangeString, svmExpoString, svmLinear, svmLinear2, svmLinear3, svmLinearWeights, svmLinearWeights2, svmPoly, svmRadial, svmRadialCost, svmRadialSigma, svmRadialWeights, svmSpectrumString, tan, tanSearch, treebag, vbmpRadial, vglmAdjCat, vglmContRatio, vglmCumulative, widekernelpls, WM, wsrf, xgbDART, xgbLinear, xgbTree, xyf'"
      ],
      "text/latex": [
       "'Supported Models: ada, AdaBag, AdaBoost.M1, adaboost, amdai, ANFIS, avNNet, awnb, awtan, bag, bagEarth, bagEarthGCV, bagFDA, bagFDAGCV, bam, bartMachine, bayesglm, binda, blackboost, blasso, blassoAveraged, bridge, brnn, BstLm, bstSm, bstTree, C5.0, C5.0Cost, C5.0Rules, C5.0Tree, cforest, chaid, CSimca, ctree, ctree2, cubist, dda, deepboost, DENFIS, dnn, dwdLinear, dwdPoly, dwdRadial, earth, elm, enet, evtree, extraTrees, fda, FH.GBML, FIR.DM, foba, FRBCS.CHI, FRBCS.W, FS.HGD, gam, gamboost, gamLoess, gamSpline, gaussprLinear, gaussprPoly, gaussprRadial, gbm\\_h2o, gbm, gcvEarth, GFS.FR.MOGUL, GFS.LT.RS, GFS.THRIFT, glm.nb, glm, glmboost, glmnet\\_h2o, glmnet, glmStepAIC, gpls, hda, hdda, hdrda, HYFIS, icr, J48, JRip, kernelpls, kknn, knn, krlsPoly, krlsRadial, lars, lars2, lasso, lda, lda2, leapBackward, leapForward, leapSeq, Linda, lm, lmStepAIC, LMT, loclda, logicBag, LogitBoost, logreg, lssvmLinear, lssvmPoly, lssvmRadial, lvq, M5, M5Rules, manb, mda, Mlda, mlp, mlpKerasDecay, mlpKerasDecayCost, mlpKerasDropout, mlpKerasDropoutCost, mlpML, mlpSGD, mlpWeightDecay, mlpWeightDecayML, monmlp, msaenet, multinom, mxnet, mxnetAdam, naive\\_bayes, nb, nbDiscrete, nbSearch, neuralnet, nnet, nnls, nodeHarvest, null, OneR, ordinalNet, ORFlog, ORFpls, ORFridge, ORFsvm, ownn, pam, parRF, PART, partDSA, pcaNNet, pcr, pda, pda2, penalized, PenalizedLDA, plr, pls, plsRglm, polr, ppr, PRIM, protoclass, pythonKnnReg, qda, QdaCov, qrf, qrnn, randomGLM, ranger, rbf, rbfDDA, Rborist, rda, regLogistic, relaxo, rf, rFerns, RFlda, rfRules, ridge, rlda, rlm, rmda, rocc, rotationForest, rotationForestCp, rpart, rpart1SE, rpart2, rpartCost, rpartScore, rqlasso, rqnc, RRF, RRFglobal, rrlda, RSimca, rvmLinear, rvmPoly, rvmRadial, SBC, sda, sdwd, simpls, SLAVE, slda, smda, snn, sparseLDA, spikeslab, spls, stepLDA, stepQDA, superpc, svmBoundrangeString, svmExpoString, svmLinear, svmLinear2, svmLinear3, svmLinearWeights, svmLinearWeights2, svmPoly, svmRadial, svmRadialCost, svmRadialSigma, svmRadialWeights, svmSpectrumString, tan, tanSearch, treebag, vbmpRadial, vglmAdjCat, vglmContRatio, vglmCumulative, widekernelpls, WM, wsrf, xgbDART, xgbLinear, xgbTree, xyf'"
      ],
      "text/markdown": [
       "'Supported Models: ada, AdaBag, AdaBoost.M1, adaboost, amdai, ANFIS, avNNet, awnb, awtan, bag, bagEarth, bagEarthGCV, bagFDA, bagFDAGCV, bam, bartMachine, bayesglm, binda, blackboost, blasso, blassoAveraged, bridge, brnn, BstLm, bstSm, bstTree, C5.0, C5.0Cost, C5.0Rules, C5.0Tree, cforest, chaid, CSimca, ctree, ctree2, cubist, dda, deepboost, DENFIS, dnn, dwdLinear, dwdPoly, dwdRadial, earth, elm, enet, evtree, extraTrees, fda, FH.GBML, FIR.DM, foba, FRBCS.CHI, FRBCS.W, FS.HGD, gam, gamboost, gamLoess, gamSpline, gaussprLinear, gaussprPoly, gaussprRadial, gbm_h2o, gbm, gcvEarth, GFS.FR.MOGUL, GFS.LT.RS, GFS.THRIFT, glm.nb, glm, glmboost, glmnet_h2o, glmnet, glmStepAIC, gpls, hda, hdda, hdrda, HYFIS, icr, J48, JRip, kernelpls, kknn, knn, krlsPoly, krlsRadial, lars, lars2, lasso, lda, lda2, leapBackward, leapForward, leapSeq, Linda, lm, lmStepAIC, LMT, loclda, logicBag, LogitBoost, logreg, lssvmLinear, lssvmPoly, lssvmRadial, lvq, M5, M5Rules, manb, mda, Mlda, mlp, mlpKerasDecay, mlpKerasDecayCost, mlpKerasDropout, mlpKerasDropoutCost, mlpML, mlpSGD, mlpWeightDecay, mlpWeightDecayML, monmlp, msaenet, multinom, mxnet, mxnetAdam, naive_bayes, nb, nbDiscrete, nbSearch, neuralnet, nnet, nnls, nodeHarvest, null, OneR, ordinalNet, ORFlog, ORFpls, ORFridge, ORFsvm, ownn, pam, parRF, PART, partDSA, pcaNNet, pcr, pda, pda2, penalized, PenalizedLDA, plr, pls, plsRglm, polr, ppr, PRIM, protoclass, pythonKnnReg, qda, QdaCov, qrf, qrnn, randomGLM, ranger, rbf, rbfDDA, Rborist, rda, regLogistic, relaxo, rf, rFerns, RFlda, rfRules, ridge, rlda, rlm, rmda, rocc, rotationForest, rotationForestCp, rpart, rpart1SE, rpart2, rpartCost, rpartScore, rqlasso, rqnc, RRF, RRFglobal, rrlda, RSimca, rvmLinear, rvmPoly, rvmRadial, SBC, sda, sdwd, simpls, SLAVE, slda, smda, snn, sparseLDA, spikeslab, spls, stepLDA, stepQDA, superpc, svmBoundrangeString, svmExpoString, svmLinear, svmLinear2, svmLinear3, svmLinearWeights, svmLinearWeights2, svmPoly, svmRadial, svmRadialCost, svmRadialSigma, svmRadialWeights, svmSpectrumString, tan, tanSearch, treebag, vbmpRadial, vglmAdjCat, vglmContRatio, vglmCumulative, widekernelpls, WM, wsrf, xgbDART, xgbLinear, xgbTree, xyf'"
      ],
      "text/plain": [
       "[1] \"Supported Models: ada, AdaBag, AdaBoost.M1, adaboost, amdai, ANFIS, avNNet, awnb, awtan, bag, bagEarth, bagEarthGCV, bagFDA, bagFDAGCV, bam, bartMachine, bayesglm, binda, blackboost, blasso, blassoAveraged, bridge, brnn, BstLm, bstSm, bstTree, C5.0, C5.0Cost, C5.0Rules, C5.0Tree, cforest, chaid, CSimca, ctree, ctree2, cubist, dda, deepboost, DENFIS, dnn, dwdLinear, dwdPoly, dwdRadial, earth, elm, enet, evtree, extraTrees, fda, FH.GBML, FIR.DM, foba, FRBCS.CHI, FRBCS.W, FS.HGD, gam, gamboost, gamLoess, gamSpline, gaussprLinear, gaussprPoly, gaussprRadial, gbm_h2o, gbm, gcvEarth, GFS.FR.MOGUL, GFS.LT.RS, GFS.THRIFT, glm.nb, glm, glmboost, glmnet_h2o, glmnet, glmStepAIC, gpls, hda, hdda, hdrda, HYFIS, icr, J48, JRip, kernelpls, kknn, knn, krlsPoly, krlsRadial, lars, lars2, lasso, lda, lda2, leapBackward, leapForward, leapSeq, Linda, lm, lmStepAIC, LMT, loclda, logicBag, LogitBoost, logreg, lssvmLinear, lssvmPoly, lssvmRadial, lvq, M5, M5Rules, manb, mda, Mlda, mlp, mlpKerasDecay, mlpKerasDecayCost, mlpKerasDropout, mlpKerasDropoutCost, mlpML, mlpSGD, mlpWeightDecay, mlpWeightDecayML, monmlp, msaenet, multinom, mxnet, mxnetAdam, naive_bayes, nb, nbDiscrete, nbSearch, neuralnet, nnet, nnls, nodeHarvest, null, OneR, ordinalNet, ORFlog, ORFpls, ORFridge, ORFsvm, ownn, pam, parRF, PART, partDSA, pcaNNet, pcr, pda, pda2, penalized, PenalizedLDA, plr, pls, plsRglm, polr, ppr, PRIM, protoclass, pythonKnnReg, qda, QdaCov, qrf, qrnn, randomGLM, ranger, rbf, rbfDDA, Rborist, rda, regLogistic, relaxo, rf, rFerns, RFlda, rfRules, ridge, rlda, rlm, rmda, rocc, rotationForest, rotationForestCp, rpart, rpart1SE, rpart2, rpartCost, rpartScore, rqlasso, rqnc, RRF, RRFglobal, rrlda, RSimca, rvmLinear, rvmPoly, rvmRadial, SBC, sda, sdwd, simpls, SLAVE, slda, smda, snn, sparseLDA, spikeslab, spls, stepLDA, stepQDA, superpc, svmBoundrangeString, svmExpoString, svmLinear, svmLinear2, svmLinear3, svmLinearWeights, svmLinearWeights2, svmPoly, svmRadial, svmRadialCost, svmRadialSigma, svmRadialWeights, svmSpectrumString, tan, tanSearch, treebag, vbmpRadial, vglmAdjCat, vglmContRatio, vglmCumulative, widekernelpls, WM, wsrf, xgbDART, xgbLinear, xgbTree, xyf\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models <- getModelInfo()\n",
    "sprintf(\"Number of models: %d\", names(models) %>% length)\n",
    "sprintf(\"Supported Models: %s\", names(models) %>% paste(collapse = \", \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**WOW** 237 models! Note that *not all these models* are natively implemented in **CARET**. Indeed, the majority of them is developed in third-party **R** packages which you must install before using the corresponding models. The result of $\\texttt{getModelInfo}()$ can be used to find which package you have to install to use a particular method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Linear Regression'"
      ],
      "text/latex": [
       "'Linear Regression'"
      ],
      "text/markdown": [
       "'Linear Regression'"
      ],
      "text/plain": [
       "[1] \"Linear Regression\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "NULL"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models$lm$label\n",
    "models$lm$library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'k-Nearest Neighbors'"
      ],
      "text/latex": [
       "'k-Nearest Neighbors'"
      ],
      "text/markdown": [
       "'k-Nearest Neighbors'"
      ],
      "text/plain": [
       "[1] \"k-Nearest Neighbors\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'kknn'"
      ],
      "text/latex": [
       "'kknn'"
      ],
      "text/markdown": [
       "'kknn'"
      ],
      "text/plain": [
       "[1] \"kknn\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models$kknn$label\n",
    "models$kknn$library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if we want to use *k-Nearest Neighbors* model we have to first install the **kknn** package (in the binder image we've already installed it!). On the contrary, if we want to use *Linear Regression* we don't have to install anything (**CARET** uses the standard R function **lm**)\n",
    "\n",
    "___\n",
    "\n",
    "#### Ok nice\n",
    "but how can I fit a model with **CARET**?\n",
    "\n",
    "Easy. There are just **three** fundamental parameters:\n",
    "> - $\\texttt{x}$: observations\n",
    "- $\\texttt{y}$: labels\n",
    "- $\\texttt{method}$: which model you want to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear Regression \n",
       "\n",
       "32 samples\n",
       "10 predictors\n",
       "\n",
       "No pre-processing\n",
       "Resampling: Bootstrapped (25 reps) \n",
       "Summary of sample sizes: 32, 32, 32, 32, 32, 32, ... \n",
       "Resampling results:\n",
       "\n",
       "  RMSE       Rsquared   MAE      \n",
       "  0.9515276  0.7614885  0.7143102\n",
       "\n",
       "Tuning parameter 'intercept' was held constant at a value of TRUE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = mtcars %>% select(-cyl), \n",
    "      y = mtcars$cyl, \n",
    "      method = \"lm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model output is quite verbose, here we'll focus only on the *last line*...what is a **tuning parameter**? In a nutshell, it's a parameter on which **CARET** will try to optimize the fitted model (we'll see more on this topic in the second lesson)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each model supported by **CARET** has its **tuning parameters**. To see which parameters a given model supports, you can explore the output of $\\texttt{getModelInfo}()$. Here you have the **tuning parameters** of the *Linear Regression*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>parameter</th><th scope=col>class</th><th scope=col>label</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>intercept</td><td>logical  </td><td>intercept</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       " parameter & class & label\\\\\n",
       "\\hline\n",
       "\t intercept & logical   & intercept\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "parameter | class | label | \n",
       "|---|\n",
       "| intercept | logical   | intercept | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  parameter class   label    \n",
       "1 intercept logical intercept"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "models$lm$parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only one parameter ($\\texttt{intercept}$), with a logical (*True* or *False*) class that controls (as the name suggests) the value of the intercept in the fitted model. To set the value of a **tuning parameter** you can use the $\\texttt{tuneGrid}$ parameter as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = .outcome ~ ., data = dat)\n",
       "\n",
       "Coefficients:\n",
       "(Intercept)          mpg         disp           hp         drat           wt  \n",
       "  12.107199    -0.004857     0.004610     0.003723    -0.427435    -0.222489  \n",
       "       qsec           vs           am         gear         carb  \n",
       "  -0.187945    -0.644076    -0.500770    -0.500323     0.179872  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = mtcars %>% select(-cyl), \n",
    "      y= mtcars$cyl, \n",
    "      method = \"lm\", \n",
    "      tuneGrid = data.frame(intercept = T))$finalModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = .outcome ~ 0 + ., data = dat)\n",
       "\n",
       "Coefficients:\n",
       "      mpg       disp         hp       drat         wt       qsec         vs  \n",
       " 0.027136   0.008235   0.009147  -0.008767  -0.463077   0.276909  -1.401723  \n",
       "       am       gear       carb  \n",
       "-0.452727  -0.233211   0.221562  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = mtcars %>% select(-cyl), \n",
    "      y= mtcars$cyl, \n",
    "      method = \"lm\", \n",
    "      tuneGrid = data.frame(intercept = F)\n",
    "     )$finalModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that selecting the $\\texttt{finalModel}$ element of the $\\texttt{train}$ function result you can obtain the fitted model description.\n",
    "\n",
    "Quite often, the **tuning parameters** supported by **CARET** are not all the possible parameters of the underlying fitting procedure. If you want to use a parameter not covered by **CARET** you can *pass it* directly in the $\\texttt{train}$ function. As example consider the $\\texttt{subset}$ parameter in the $\\texttt{lm}$ function, you can provide it to **CARET** as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = .outcome ~ ., data = dat)\n",
       "\n",
       "Coefficients:\n",
       "(Intercept)          mpg         disp           hp         drat           wt  \n",
       "  12.107199    -0.004857     0.004610     0.003723    -0.427435    -0.222489  \n",
       "       qsec           vs           am         gear         carb  \n",
       "  -0.187945    -0.644076    -0.500770    -0.500323     0.179872  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = mtcars %>% select(-cyl),\n",
    "      y= mtcars$cyl,\n",
    "      method = \"lm\"\n",
    "     )$finalModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "Call:\n",
       "lm(formula = .outcome ~ ., data = dat, subset = ..1)\n",
       "\n",
       "Coefficients:\n",
       "(Intercept)          mpg         disp           hp         drat           wt  \n",
       "    9.63646     -0.54925      0.02618     -0.03921      3.51626      0.17568  \n",
       "       qsec           vs           am         gear         carb  \n",
       "   -0.08000      0.08408      0.82914     -1.41601           NA  \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = mtcars %>% select(-cyl),\n",
    "      y= mtcars$cyl,\n",
    "      method = \"lm\",\n",
    "      subset = 1:10 # here you pass the additional tuning parameter!\n",
    "     )$finalModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "## TOY Datasets\n",
    "\n",
    "In the following lessons we are going to use some simple datasets, that have been developed to challenge the simplest classification models.\n",
    "\n",
    "These datasets will be generated leveraging some of the functions provided by the **R** package [**mlbench**](https://cran.r-project.org/web/packages/mlbench/index.html).\n",
    "\n",
    "The full code to generate these datasets is contained the *.R* source file *src/lib.R*. Note that to correctly source this file you should set correctly the working folder:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load all the datasets using the $\\texttt{get}$*_*$\\texttt{full}$*_*$\\texttt{dataset}()$ function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>x</th><th scope=col>y</th><th scope=col>class</th><th scope=col>type</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>0.02061427</td><td>0.2030190 </td><td>class_2   </td><td>normal    </td></tr>\n",
       "\t<tr><td>0.25061022</td><td>0.2275554 </td><td>class_2   </td><td>normal    </td></tr>\n",
       "\t<tr><td>0.04395160</td><td>0.1796892 </td><td>class_2   </td><td>normal    </td></tr>\n",
       "\t<tr><td>0.77235720</td><td>0.8031989 </td><td>class_1   </td><td>normal    </td></tr>\n",
       "\t<tr><td>0.82086031</td><td>0.6966486 </td><td>class_1   </td><td>normal    </td></tr>\n",
       "\t<tr><td>0.70940915</td><td>0.8332838 </td><td>class_1   </td><td>normal    </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       " x & y & class & type\\\\\n",
       "\\hline\n",
       "\t 0.02061427 & 0.2030190  & class\\_2  & normal    \\\\\n",
       "\t 0.25061022 & 0.2275554  & class\\_2  & normal    \\\\\n",
       "\t 0.04395160 & 0.1796892  & class\\_2  & normal    \\\\\n",
       "\t 0.77235720 & 0.8031989  & class\\_1  & normal    \\\\\n",
       "\t 0.82086031 & 0.6966486  & class\\_1  & normal    \\\\\n",
       "\t 0.70940915 & 0.8332838  & class\\_1  & normal    \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "x | y | class | type | \n",
       "|---|---|---|---|---|---|\n",
       "| 0.02061427 | 0.2030190  | class_2    | normal     | \n",
       "| 0.25061022 | 0.2275554  | class_2    | normal     | \n",
       "| 0.04395160 | 0.1796892  | class_2    | normal     | \n",
       "| 0.77235720 | 0.8031989  | class_1    | normal     | \n",
       "| 0.82086031 | 0.6966486  | class_1    | normal     | \n",
       "| 0.70940915 | 0.8332838  | class_1    | normal     | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  x          y         class   type  \n",
       "1 0.02061427 0.2030190 class_2 normal\n",
       "2 0.25061022 0.2275554 class_2 normal\n",
       "3 0.04395160 0.1796892 class_2 normal\n",
       "4 0.77235720 0.8031989 class_1 normal\n",
       "5 0.82086031 0.6966486 class_1 normal\n",
       "6 0.70940915 0.8332838 class_1 normal"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "full_df <- get_full_dataset()\n",
    "full_df %>% head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can plot its content using **ggplot**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAADFBMVEUAAAAAAP//AAD///9D\npfB4AAAACXBIWXMAABJ0AAASdAHeZh94AAAgAElEQVR4nO2d12IkKQxFa2b+/5933e5UmSDg\nSpzzsDu2O4DQRaHS8g8AqllGDwAgAggJwACEBGAAQgIwACEBGICQAAxASAAGICQAAxASgAEI\nCcCAIELaTiPItAKxBF+UoJMLOi3HRF+RoPMLOi3HRF8R7/Nb/uffM3H4+dfn59c/3/+HkSzv\nRVotynu9vK9RhOE/12jZ/Pz5p/tpBmC9SNvF8S4j9x62fP6/nPzsfIZRuNjdQiyS8xlcCWl5\npQ3O5xiDjZCWdW7nf4mcz+AuIj1/cj7LCBxEpOe/iUgCpAnJ/TQDcCykOPm38xkcNBdoNkhy\nvCiv1Nv/+nifwXf7e/0z7W8pNrvd1+IQkQDgCUICMAAhARiAkAAMQEgABiAkAAMQEoABCAnA\nAIQEYABCAjAAIQEYgJAADEBIAAYgJAADEBKAAQgJwACEBGAAQgIwACEBGICQAAxASAAGICQA\nAxASgAEIac9c98Gba7bNwIg7lr9//85jlrlm2w5suOXHs+bxrZaznSrWTTTVRBCS4UfPYkaE\ntAchqX+yJNNMNJ3JdtKWAen4o0OmfAGnVE3IhT7lZrbFxjgVUsyNKt6MwJIKrz95a9CUL9yE\noIpNAKry+uNghpAgPtso0sDrERKEZ+fkLbyeGgmis9dNC68P2cwJOKU3IResKQcBSM+IeiP6\nQXFMRsRMIdriwGaiQxw+pGb7S9CitjGa2/0Xqss6ekTt9pcMi8t7D7xBSMdf3/acybSPFk0W\n4AiEdPz1Lc/iz9CR4tLAIaLbXmAhpWZs0whpVAZr/L2aifjoMQnsL7MIaZSpBZa4A8NnKLC/\nTLLSg/aLnO8VcIZS3A7cEsfrl8bPBB0IyfOO5nXckMHDQfWF5DrHdjrs0biKYU8Hla+RENJ0\n+MpBXg6q3rVDSLPhbMXdDNfX/rTG67iH4sYzn7hxUFcZ8xq3Ax+JNyF5dlAvYOAS3Gzx0Avc\noQi2eFiDPwCck7xjIiSAU9JzeN9Cap1hkcHNTc7pTa3H0pLWNT89hcmZREitu9DuutxgDEJy\n8fkgzxw1EkKC1szRtaNGAhV8+wldOxDBpaPg36CGR48k4wI5tBwyKdTQAwA9pPwxLdQgJNBD\nyR8TFYKQpsJJQaw0yFSFUCNZI+ysXhZbaYzJoUZ43V0i7Kxu0g+pIQovaGSUnVV5bCu0hkio\nGUEnZy1aXIQEbujjrIXphpcsxcMYoTFmznoRdIrV6iRLcTFIaIyRs14J0k2OVkjcmUFvLrWC\nkADSuNaKl2KnkMBTg87cBB0nxU4hPecW25IQPehc0nHeM5t5EibeKusmnmO46NWmLBN7d0eq\nbJwVYxDSGJY/f/5cWj2S0E7m0mGKNV+QJw2ENIQfHV0qKVLGfTKXHlPsJ6RQK+aHOyFF2t9O\n5tJlih2FFCqHcANC0hcSMcYDNzUSQjL67rp3E2PU2C/JzSJF2g2d1kigx22P7uAtgXZDn127\nafDjau+KyM+Qg4C573GU/LyEVBCYoAqsfYuncvwppNuDR2ANxr7Fk5CeoUhLSFOkmSJTVLa1\nKyH9WlJKSI4y4wo0Zqhta+3RHSJUI/nah4qRmKC6rZXj5Qk6Q1ZfXCMkJjiJredkksWVmOAk\ntp4Uh5lxARoznMPWs7IsOolmM0QmOIGlJ2aGfTL6/GA8U2TuwacHAlQJyUuu4mOU4JkaIbnJ\nCl0MEnxTrgY/WaGHMYJ3ivMzhARgAEICsIAaCfrgpatVipf5+RglnCF0mvfcsAiukbrwaGpY\nA9cgJBVYA9cgJBVYA9+k1UheKnbHYGDnpGiEjkR7sG98yP86gHnjE1VIUgmr0FCgEUGFpHXS\nw8lIpMQOlYSskcROwzseiJbYoZaI+6IHIYmNEWCPmJMiJHCKVtqEkMArUgkrNRKAAXTtAAxA\nLwAGICQAAxASgAEIqQIqSXiR4wn4zRp6m/AmwxHwmzUcbYMP6X6A32zAIPABIRWDQcJRUbwg\npHLIdYNRs6DUSBXQfclA31h1z3HKea28KUAWB9twNyEBlOKhMEBIII8HIa2jZmb+JT41CIIL\nIX2LJzcVVZ8aBMFBjfRNtvAdzQ1c46tVhZAADDSLkNrha0udGJMskhqpFc6S/Hkx6mvQtWuD\nj7YTjFopPCMRhOSF50p1zsS9e8aBudpYECG54ZGE987EnXvGwe3hW90xnhrJDT/RqPe+59s1\nDh5Y0u4ZJnTtHIGQsugqJFfMrnqElAVCOlFMyCciZUGNlMP/XtStRtLkeLaz7SZHNIzJRx/t\n2tgPL+rUtdPkRDEIqSWHwc6zsX26i6nOEVJ/jssvz8Z26S62ufuZCRTz2yiZAkJSwLqbdKYY\nPa8NcxwunpAu990ETxrhbMvPuSuWDqWnmGNEzwwpMV+4GunKDAm5zZD05/fUFd9WL0JTSGVh\nMlzX7pyErG9MYqjpTx2QnLjhoMRmZkVfIWXkB5L+1AXFGgkh3XGkko272wkpx0XmFZJiNYeQ\nbtkXQLvfWNVIeauhuDHPi91qhF3T7f53dFqezRaZua0JbswTY7Ya0yxqw8sr3GdraLueaSzY\nsEnnPVtTPA3CHfMYsKG7+N7RPZ4gosdE9vPt7u1ASBZgv+lBSBb4tR8BxgpqJAPcGjB99VHc\nHVioHq8WTM9H2G+hA15dLFlIVADQgwsPkw74CAmkOPcw8cOMaRnb4zZDB6+U3iTAIaf+JH/i\nS4oWHmo70hFBCmwpFpKHPf0ZjXYjJd1rgQePaEepkMQTv1+u7rGDkIyZPMoX1kjyid8DhNSP\n2W1a2LXzIaTzm1WVrPncqcsdNkLya+PCcTsR0unCFCxY89TFrxP9YCKk4yzIhWFKh+iiRrKk\neeri3aIGG83JrRddGKZ4hL0f0Tl6V2otJC8x/pz6JTq0gRPDyA/wl/EtIYTUHoTUGoWWUGMt\nO/GXthxlcU4MIz/ABwpCap1d+igFGnN4M2AXhtEf4Q8SQmrN8DJQFReGcTDEH8bXSABXyHvn\ncztysSvBvKi7J6EIXCDupFMURxAAbR89uSwPQA1pH13+IKR0KCNHIm375c/jUnGLMa69LKLP\nUU0ORdr0j8zOxOfXXhbR5+Rid8TN6gLpyZo5x/qD5HzOArVJZWxWISQnPYX3PReWK2snrANC\n6k3GcHycAnSH9Ayeq/EQ0+mypGx9EwjpxA6jdvuMO+H6OCn1Du0JvG6nddG9S1ux+DXSyQmf\no2aKkLRYnnd4rBXSBF27IwbG3mQJI6RO2AhpTkYaJ3mzokbqhEGNdPzG+GHJxy4TYiE8TKG+\na3f4NgcuVs0Uk5TAm5XNdq/l2Vm3+TRZ4s9QBGdmttthz+6vPyXIrRpfBjTM+ZerDsZksKHU\n48t+lq7PqeUv5raDUTT2ZT7TJe9ysZOHpOnWDh4mUYpV892ZhWyTkPYpjYtjJHdCipz5mR0O\n9mYg282x9Vbr5Kj9tVJGZX5d4uC0QvJF7jINO8X06nsHCalPMEdILshcJs1EcIyQegXzSWsk\nb2Qtk2oiOKRGKjBGWTyfsmvnj5xlUhXSkIwz3xhj4/lMVwOrIyukXEy8JVcXg62X9sWayXs4\ngpjZKBXMlKMHIYXZKtUJEfgHdvkQEjRhiDCHHXfSr5EQkk9KciyDJ8GOOnVvaDynRopLiUdb\n1DeRTyk6ha5dXAqEZBNNZvSW+WZ8QTAHGCakGcFmH8KlJPkTQkilYLM37Z2oe8TL/8Jwm0kv\npjba5q6RrYXkomUTLL3txsxW2+y+rYXEQYQYHG81E6/rTjiN0xqEFIKTvGLidd1HoLZpDUKK\nwNkqTryu3TtULmokuKZUSJFrz+4dqsjGnIVCIZ1tog4auaM+FGJTVCOdyq86Scm92ITDGyBC\nSdfuREj1ZXO6MH4fRcEBd9BmjJAyHoz4eF3zQzzkeA2ZwrpFNVI/Ib2ea95KSL9rTOLYkjms\nW9a1q62RcoXUajH6xLu5CWDdlJBaOMHaaJ0qjPcqtOnaNY538C+CkJLCxqgJpgqjaV6AkDrg\n3rpphYz8BA8U9/pVdZRqnDjCA5Or1wf2K4IIac9rYW6eoZBi+vdHzdBXGobB/VSG3mjr58tv\np+DPgb7DyLmSErdBFOSB0bes+5sgZGE/OnHyJCG5T8y7cb2VaGw0o8+bT/l+BTsdcxZTEJIl\n988Yu+9XNRcbQqrgXAopNVILIWnszrbc58e3VuzRqRl8AUpQISV17eyXN2Rrr15IfWL/4E3M\nc410H5Fu3m9s+pjJohchjcZh1+495LsaqTNBHaa6Rgpql2zULPC1dDddu85EdZjqrl3IlDcf\nMRMkuKu9R6dlgTjMCSvzRezIJJHkQv2Mk5qVGw5o+TnXTswMbpl3t0k5+t+x95jWcLX06MfS\n+119LXVHzX8TSGh/dT0a1ntLey6918UXiwAI6eoVfQ8rd95ifQtJzXHVxtMROSF9fXGxpDLe\n+bokqfCbBiPnuGIRsiNiNdL31xavyeedqd1br2svJySxmq0jsu2qch/5vFPknMuG3z9vBFBD\ndhUMhHT4EaN1s6Uy3qtNZ1pkl6GRkNS28NFXCIARuktYXyMdCEmuqEBIQRBewsKsZXnw+8/D\ngISQwJ5wS7gSz+YhsYvgnbd4aJI+DW8QKcvV9YB/nvew05oz7QJ1pG8Q2Yrr6wF/lITfQhZh\n72t3ya2QxOLRKNhPrvmyz5xCOm/2IaQvutyxxLFWv7O5SYV0vn7OL5iwJKvnUto9latG01lr\nZ4IaKW+Nky/hi05W87JQEHr90Qw2QSh81y57jT1nG3YsfzK8vFQQkYSU9JZWY+mA67UayPJI\ncTMC0nRC2mdzt1uw15n+ULZWRKXfA2qpFVKpIDzXSDsvuS+T/E61cI19r68NWTYoNligHSsh\n1bOY6zCLFayx74zDiqwVCySIUvoIaeDZYvlrPE5I+KNfugip2/nLJp44TEicnOqZHjVSLyEZ\nFTeDaiQul/BNh65dJxcxCyVjUiyEFBw3NZLzLoGokCjcrHDTtXMuJM0aiWMBZvgxo+dF/9lq\nBDd/75uTEo6sKOiJiUhGo38IyRKs2B7R+ggh7ajYq7Fie2SF5DpdbkBN5oAZ26MrJMfpcgOq\n1gk7dkC1RoIVCEkeNn4PICQAC6iRACygawciTJvETjptaMO8/fQ5Zw1tmPgI75SThkYgJAAD\nEBKABdRIABbQtQNIZVq1XIBFIJd587cLMAhkMnFH4QLsAZkgpCMKHjNAgjw3COmIbHsoXVuD\npodAjXRA9r2zha72ZEEHwQa2x7GQSDFAB4TUC7bx0DiukXwJScdu0ALPXTtPNZJQJIcWuF5a\nHU3fMkRIjuzjHizdhxFC8hSx3YOhO9G/RvJVQ3oHO/eie56FkHqCncOCkHqCneNCjdQRDB0Y\nunb9wNIABiAkAAMQEgSnT4KLkCA2nVouCAlC0+sgAEKC0CAkGMVBUeG3kY6QYBAHRYV1ndFT\nl9RIMISDLdx6V+97ygVdOxhBeyGFPAkw2HSgGoRURLDpQD3NaySEBFPQvGvn4LT07AmLzwdC\nIt9Nz7+eWXxCAAMouMMGQgLYgpAgE/kkawgICfJ4lP1FPhBbgdRIkMOzEV3gBJvGWzhZ0bWD\nDJ5Cyu9Fbw4FOehnt2by6U+OkZBCHmHNZO7ZT88jlBRoACFtaTj7cHnzNT6n+/+oiySwbDM7\nhNTsk+d6jMmg6dbLt/ATVm+jRmonpMkeCDRouioevCw+A7IdTWb/Y1SE1ONbLXIqEwmoKHoY\nLSb/yHIQUo9vLT4OtP6M6nFTJTWY+9OnvoqGGcL+kBrJQEg2EkBI7YT0kc8cbYchu8VSdhho\n8wlthDTD9vmhoZDOfjb8pqmW6pCf9nV1Zvd1RKjUovsEcbKqqVmNtP6xiZDmiHQ3VG8m3w5f\n4fzbcQxM9sbkBk0+dDWTVkKarJ3RjM9qWTr/OCGNCYVdbvnVLiAhJEuGC8kilgxScJcvbBNr\nEZI1pj5YEBj0O/HnnuzZD6mRrKny5F2VdLF9Hv5NvxN/4XGuHZGunTUVFs0R4fFrjSTQrka6\nyoHwRDAhRwUnr7WKJc32V4QEzTEQkvyhJ4QEzbEQknyuHrVGAiHqayQHxOzagRQ54UQ99OQT\nbT4AQ5hISI12wXibKxQwjxM0PFFpHiPCGdP4QKMTijhPCR5M4wIICVoyjQsgJDjD5KRzg3H4\ngBoJjrE56dxiJGY07YDRtYMjjE46NxmLEezu0J94QqLegAEgJAALwtVICAmGEK5rR40EXtHy\nWzpg4BQcFyCHk80eITmGAN6fs/Ij6Eq0cjEp13V7naljThtiMReiVddCqhvCo1QGMJeQjPro\nu/Cj1Z9HSANASGWfsux+g5DmJkCNlF6fmDj8wYdoCUmiRpIqGrvgvmt3ECDO74105+8J63+k\nGqkaScGLFbScSltrebHCgV9fefWN0VIEcRh+xruuFJ6yy8aa92GEfwd+XZFnpb1VLPwo4khI\nrYfqwgg/9BcS4ecWhPT5/GafbM02QLQXEtzip0ZCSG92D7IqFwNZmxV+ojY10ik1T8Vys/5g\nBV07AHkQEoABCAnAAIQEYABCAjAAIQEYgJAADEBIAAYgJAADEBKAAQgJwACEBGAAQgIwACEB\nGICQAAxASAAGICQAAxASgAEICcAAhARgAEICMAAhARiAkAAMQEgABiAkAAMQEoABCAnAAIQE\nYABCAjAAIQEYgJAADEBIAAYgJAADEBKAAQgJwIBZhDTLPF2wXYwIixNhDinMMk+XRFicCHNI\nYZZ5uiTC4view/J65vvy+/+fn1+/3P4FxvG1PM/FeKzU9+I8/+kW14P/91mW5/+Xf6v/fP1l\n3CBhvTyrf75/fr/OKZ7HvpHJajlWC4OQxrJ8/3/ZrMf2Z6f4nkCKkH5zBt/z9M4za9sL6bM4\nvhM77w6WIKTvFA+GcZjKrRfHt5RcDz5BSNRIKix7IW0Xx/MieR57qpBI7QZz1Gx4/X6h2SDA\n13Kscof1X4hIozlqfz9//9ntSO0AkojsbJHnBmJEdrbIcwMxIjtb5LkBdAMhARiAkAAMQEgA\nBiAkAAMQEoABCAnAAIQEYABCAjAAIQEYgJAADEBIAAYgJAADEBKAAQgJwACEBGAAQgIwACEB\nGICQAAxASAAGIKQbnN9uzRWebe135H1Y/v79i436sPz588etrd0OvAM/N//8+xcldeJHR36V\n5HXcHXgEI4TUDYQUk6eGEFIvEFJMPkLCRn2gRgrJU0ieO0ne8GxrvyNvDsEI0sFTzvG8QUJn\ncBUAAxAS3EJovgcLwR2uu2m9wEBwg+/jO73APnADQkoB+9gSsJxASClgH1NCHnuiRkoAA1my\nP8c1RIQKMYnGYCFLdkIKGaHgAFbZkq2QuApjGlhkU5Z9QEJIU8Ai27IuJxDSNLDITaFGqsRN\nn8PJMN3ixhE6kWmP8867mmG1RgPByTwkdX4sWO7YltRgIDi5J0mcvl7vbAulsUB0EBIYo5bi\n9yHb/88yOIQED2bt5mWXNmcbDjUS/Jv5+JJZJFYL6VqjmQVlIal5qBMwWgm1zjZcSOcTkMuZ\nDGm5R0S1WQnJdq6vcAbXSBfHOeWqeDua7hExTVZEsnNbxJOhCdSFWgILqe3UQpqsiHR5DE/M\nakFIDT690ef6AyG9/+Z4ahcgpD5kyMP9UaArtcTt2lEj9SFDHk6c7aI352MCxtC168OpnZ9/\naLUOzT43bJYmCIa+5xmqDiKWiQRaJYqB+waCYOdbXk8c29dQJhJo1rpASD3BzrecCslGAhWf\nch0QEVJPsPMtskK6q4GokTqCoe85q5FeEkiolK5eUpog3kecOXtzY8DSCZx17X4lsBLCse9e\na6XQ30ndlGAdaviRwCo1O1ZMm34CQlKCdajlWyUnimnUmNOpgcghEVIlyzJOSDL+q6Poccw+\n/0oeqdw6szvL7eJamhzzH0Kq49UZX75/cdhtEIkdTUBI/xBSHUEfLJaJRyGZr5Ov6avh/sok\nG/zVSPYj9jV/OWIXP8l4i8MNYqgvA+iR4kLe3Cw+CULKXTSWOIUqKRC15Eg4uyr7lrCVQ5qC\nKilQRwmScr5vnpJY4HtSpHBxVTdCEuQmx0BILUiQwkXMQkgOQUgtuJDC67zwK7F8n/lA28EJ\n1EjWvE7wPtbR8w/XUectn/3noCxV6NrZslxduvfWT1r6dnyNLStQgtwOJDYcNW5CzfuvSYrY\nfRjlUyl651JojUaOVCGlHZhFSEYInt0nNRg9blw9MzXbvhwhFYKQ3HEjle8rKApOFqJGKgMh\n+eNSH5tr+vKNKVczO4EaKRTL57K+sFmaqNTlhiU2HF98XWgeVUgkn4lgpAqWXx29bm5ncNdV\nuX026P5gDzYq5OHzy9/VMdlqHck5LUJKBRuV8croPrc/+bkxV501Bb1WcEiiYKM7DtOtzTkN\nfz85Xs03CXqtXpAUBSPdcHsT4uctIoMKSa9sEwUrXbN17qPrJqyEFGz7n0uCM821hFcN9PXj\nsvrH5zURu3YV6B0zbcpEUy1iLaSTs1Qvr7WYFMGzeJoyz0wL+W0k7IW0ftGyS/5mByHBmu+2\ndt3dG+6+puBduvw/nwUhwRf7JO7qxYU9h1h9hn/PCYWa0TUTTTWRm0eF3zhHoZAkO981hJvQ\nHRNNNY3K0ICQfgk3oTsmmmoSmQ6wD1C/Kc3+1zc3JAzgd6spRphQFhNNNYk8BzgKX8vvqQ6Z\nl8L6r5E2M2jyIHdhos2nliwh3Twydlkdx71TknPX2t/Z5WJC/reNHcGmU0/OGl8LKeHZsoHI\nmWJEc8SajQUZoeFaSF9/jOg5GxASlHMSvpZXQKq8NYorMqaIkGDD2b2Ml+3F595LoAQyQ7mO\nPUzWRmc64dDyFi0ufLf7lmNzmjor3Y5jl5ggNtXQffsxOruWRe0MceqS/uUTQnJJxDrbEoQE\nSSCkawbYhxpJlNu7hSOkCwakvnTtJLl7fgU6usZpM8bloJW5DTlOHQWuYVGNIXebE1bcGIQ0\nJ6y4NRRBU8KSm0MRNCOsuTkIaUZYc2tI7aaEJTdmumYDAfgBRjBmNiERgH/BBsZMJqTJpnsO\nJrBmri0aIT3BBGa8ioWpiob2QnJiTheDHEPmCs4Vid60nrYXs3oY4xi+b0uXIKlgOU76LtI2\nYrgxq4MhjuH74XwpS3m04k6ykgNk4gBC8s5nBdPW8uBVMt6YjY776ozkBgdDHEOukPayceMD\ne4SG7mU38jDGMaweYJ6ymNtETsgbczkeenmmWpPjOsmPXQxyDO8VLNwUHQvp+Hk1xbPxElVq\niD4/E4o2xeXvsf/42GGPn6BW+JzpUH2YE4JNR4fnk/uOfz9gPNWYCqn//VRbC9flmnrg5iFk\n/rAUUncr2Ny77vIbmn76xIQTkmWN1NsKyXdTreinFL4P7jipkPwKybBrpyqkisDlc0ldcPIw\nCrc6sqSzFRKFVHMbcNa0mMINOly/qohkK9iYKy3UIKQRRA0tWkK3snLSrBDSADwXO1do7Q+d\nrUyN1J+gQhKbVveuBF273oh5XDEb1xGblthwLvAwRk20cqBStrNQ81w3VnYxSE20qvI7zpvx\neyUpzcuLlX2MEmo50cfRaXBynqs3ogMcDBHqOcvYBDK5W5moxchj9EcIBpwKZriX3g5AQOsp\nyA9QARe5xSXn3jh4bvcyWb1CdyVUx6XE8F3bANU55AmpzyzKruO0H0c0nOQWN4ju5QnGXVY6\nar8SZac3SFpXixhCUiUhyKxuntF8JQpPuJvTP7K2Z4TUlIy1QEhipKbazzVWrS/mo8dKIKRk\nkm9U93qVaH0xIT1WghoplYxbpxKKJoSuXSIICcyZ0k/SUm2EJIZ0hi08tIakLQlNBim0l0N3\nZAJIb4GzIZ4gyA4MYAVCAjAAIQFYoFMjHT4cYcA4AEpQKVkPj9hqDM0XKgsKQzg+hwiXyEYn\nxViDvvuAkGxQLXpV9d2Y/tsHQrJBVEj5wwoRwUZsH9RIJkQRUogINmYx6NqZoOmBuS4luh9k\nojMLhTF4QzMnytS3jgvWoDMLhTGACXn61nHBKmTSA4lBwABkXPAHu8fTWn521jg6fAdIIpSh\nthR1pw1DxpYwLy3TzF4pLEKC4SAkAAMQEoAF1EgAFmwbH5aNkIrPyrkLbOl3ADRDpDWfc6tI\ngeFCNUKdbAtEDhZn3bx4+GihHpEN3AyEBCMQ8Ts7RCaEkCZDxO8MEQmx1EhzEU9IKkUfXbu5\nENnApwb7R0BkA58ZFgDAAIQEPriIugoBefwIABK4qAMlSsThA1BFYZeDNxedSY2m5ejvV0Vi\nl4M3CMknGosDbxCSTzQWBz5QI7kEIclB184lErsciHEl5p7j8ITCLpdEv4G6MUkrrk5indw0\n/ukXOnPOhQ7J5WUVU1smAP2Kuayrc0KCkAKDkPqBkAKDkDpCjRQYaqSO0LULDF07CTANgAEI\nCcAAhARFkOetwRpQAp2HDRgDCqAXvgVbQAFzCCknfY1uCyjn6qjJDELKSl+D2wLKufSjCWqk\nvM0iuDGgmBs/it+1Q0jwoNLVp8jerkBI8EPtSXjBhZSwzVAjgcVp4aHLoKTJ0bUDi+srApdB\n9uE2rKlmh/sgXYGQIJUWFyqFiVEICZKx9/pAVZP5VILYBToQqo9nvc0EMQt0IJSQrMEskApC\nugCzQDKBaiRzsAukE6ZrZw+GATCgVkjsUfBmZmeonDlZM7yZ2hnqJk4fJzC58WVuZ0BI/mmT\nUWXHl7mdASG5p83dv/OXdm5noEbyTqPTvAtk4dQZbAI6XTvv6AjJpzMYyd/hzGFFqwuPnMaX\nXKwS0glMFZ1WT0hyGV+yQUjwYg6PbwRCArCAGgmGEyIWanTtYGIm6Uck0dEOIbYv+DD3EdgN\n/czA9hUNhPRFIzPsow9WDwdL+kUbMxxEH6weD5KMD03scCQahBQQ72Wv4fi7CYntC9SwdMl+\nQnK/fYmDeXMxTZK61UjQlrdXfOoAAAaUSURBVFZn3AXGgZDYHnvT/eETAVbYg5CgM72FFCLn\nkK+RoDudhRSkB6vetYP+9K2RggjJEGwRha5FC0La0skWAUpT+CZEjWRJY2M8BYTZw8HWuKat\nNZ4CIhGA6DT17peAEBJEByEBGHDi3UbXsb8ERI0EwTl272rH3zYZKE0HgNE7cmjq6lQMAQnA\naaw9aSIkaiIBup/GOjcIKSoIqStNaiSEJABC6kqbrh1dOgFuaySjApY6+Acu7IvLzSIYNSPY\nNB9gglkxSv3MnuZwu/Vqb87KY4OWaAnpPq6JRz7hoUFTcoV0EhAKHtp88EH3n6LewLK71FY6\n8MKevBrpNCDkRorD1yOk9+dY3kcCTXYhx84Xfpy3XCf3POwlpGa+ZXUzIsP9QjwZnhOzBT75\noE410uWHVIlMT0jqMXxOWgupT9fuchZ1SkVIkIRZnpAQe8oUc/Aooe1vrnyr0u/0aiSEpElF\nQFi/9e6DCl1p/7bj34gLybCKo0Zyz0Y5WQta6ND7t509XOgis+smpD79NLp2zlm7a6aHNhXS\nlW91rJGIFZDAxoOlhHT5Eb26dnadG0JOZOqE1LJGasgIIRHYYrP1k+xzH5p17RoyQEi05aKz\nVc6xQ8fKSwbUSAgpPCkaCZaXDOjaISSI5wUjZhJsL4IS5hXSspgltbGyYyhhWiE9wsj1xNEH\npBMsL0mdynMDuZp6umVQHPT0gh7fVC+k1yifr5ivXzMpfnbDLvdurhbS5zbfvy+410i07HhO\n/OyGfe6UWVsjLe8otDx1dGtfhBSAnotYe7dSLSGddO2WTxRasoXkJzuADR2FVBv7xIR0ejem\nlS6ShPQbxTxlB7Cln5Dqv0muRjpW0mqiaTXSj+J2D8QkPDWijWG77YIGkhXq2l3djWn1p2e9\ndDXy1xs2byQ8taHVftxr43NSUtcLKf9CkOefd6HMgbn84f/pLj62WAMh7famW1FsnzCb8h4o\nw7+QfCT9hTVSUu52/6GfT0FIjQggJBdcW/ir472SznW0LRKFjwDuEJ7J3IXrs1Cffeqj32/7\nC7v3pXz5khrkoBwM24MrG7/Onjv7w+eHnZKS1o4gNBQEZkm1kMqLG8qioZDymVIkpO9YciqH\n2w0PIY0kXBNicIAtqpG+R30mh4R7pSOkgUQT0ug6Iblr9/p5/5pTHaUoKc5KOiOYkIbvyllf\nfaKZ+8be2edR7o4jVo0kLaTv46Wvs7WTryYfPTG4I9Q2Ntzf7k+J+/wza7DkbdCT0f6WdH7C\nq32XpXrDByZF2jpDobQysl27vZAGqX70XgNnxCqz6sgS0hDVD89+4YRgjb86cmqkMSAkVRDS\nFzlduzEgJFUQ0hcOzECNpAo10ockO4xuiAj1huAbVuYNlzsAGJByQXjDIoUtDWIwVkjEOgiC\nlZASQ8v6ZTTkIApGNVJiaNneiwghgTipxYdN1y5REduXISQQJ7nDX+jF2wv+yoREjTQYej03\npB9zLjPkVgClQmIlh8IB1TsaC+lAEEU1EgzlykvY4R50F1JZ1w6GcuElxKonbWskmgQhOBdS\nz9NRxfdW067d/l3HOhK3CWw43W47CilK7LPp2j1/SZhyxtnO109Iyd+kvkkbjo6ELw7d4kSq\nkOQDF0KCI7o92DJNSPrXECIkGEpaqJlKSNRIUEBS7JtLSPIFIbglWo2EVGAM6p6XN7rs5E3j\nRkTQENb1QZ4sctsJIrfGg3bI51ydaCqko5u1QihOugDzhSmEBDUcC2l8mOqu5KY1EkIKz6GQ\nxjer+yu5bdeOGik8Ry47XEgDBtD4y+jahedgXdWF1MIVcW6wZ3SNdC2kJqNDSNCA0enHlVba\nxEuEBBG5ULJDIY3elwD2+BMSjTpQxFuNxKEj0MRZ1w4hwTx0FRI1E0SlZ41EzQRh6di1I9WD\nuHT0a4QEcUFI4B+B4rvnAKiRoAmjT+17jKHrl43fOCAew082fwxi7NcDHJC34yIkgCMyUzWE\nBHBAtjCmq5EAEsiPMALF9/ABAGyQSNVycTZcmAGFVC0Xb+OFGRBI1XJxN2AARRASgAEICcAA\nhARgAEICMAAhARiAkAAM4AaR0Jop3IAbREJjPJ6nkA/3tYO2uDxzLh+EBG1BSNUfjZAAIVl8\nNjoCaiSLD5+hXQN3TOEGE0wRoD0ICcAAhARgAEICMAAhARiAkAAMQEgABiAkAAMQEoABCAnA\nAIQEYABCAjAAIQEYgJAADEBIAAYgJAADEBKAAQgJwID/AKss0VQ82AaEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "plot without title"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "full_df %>% ggplot() + \n",
    "    geom_point(aes(x = x, y = y, fill = class), shape = 21) + \n",
    "    facet_wrap(~ type) + \n",
    "    theme_void() +\n",
    "    labs(x = \"\", y = \"\") +\n",
    "    scale_fill_manual(values = c(\"red\", \"blue\")) + \n",
    "    guides(fill = F)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to better adapt the *toy datasets* to the input/output format required by **CARET** we have prepared an alternative function to obtain these datasets, $\\texttt{get}$*_*$\\texttt{partitioned}$*_*$\\texttt{df}()$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'list'"
      ],
      "text/latex": [
       "'list'"
      ],
      "text/markdown": [
       "'list'"
      ],
      "text/plain": [
       "[1] \"list\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'normal'</li>\n",
       "\t<li>'circles'</li>\n",
       "\t<li>'spirals'</li>\n",
       "\t<li>'linear'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'normal'\n",
       "\\item 'circles'\n",
       "\\item 'spirals'\n",
       "\\item 'linear'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'normal'\n",
       "2. 'circles'\n",
       "3. 'spirals'\n",
       "4. 'linear'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"normal\"  \"circles\" \"spirals\" \"linear\" "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "'list'"
      ],
      "text/latex": [
       "'list'"
      ],
      "text/markdown": [
       "'list'"
      ],
      "text/plain": [
       "[1] \"list\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'full'</li>\n",
       "\t<li>'full_train'</li>\n",
       "\t<li>'x_train'</li>\n",
       "\t<li>'y_train'</li>\n",
       "\t<li>'full_test'</li>\n",
       "\t<li>'x_test'</li>\n",
       "\t<li>'y_test'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'full'\n",
       "\\item 'full\\_train'\n",
       "\\item 'x\\_train'\n",
       "\\item 'y\\_train'\n",
       "\\item 'full\\_test'\n",
       "\\item 'x\\_test'\n",
       "\\item 'y\\_test'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'full'\n",
       "2. 'full_train'\n",
       "3. 'x_train'\n",
       "4. 'y_train'\n",
       "5. 'full_test'\n",
       "6. 'x_test'\n",
       "7. 'y_test'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"full\"       \"full_train\" \"x_train\"    \"y_train\"    \"full_test\" \n",
       "[6] \"x_test\"     \"y_test\"    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "partitioned_df <- get_partitioned_df()\n",
    "class(partitioned_df)\n",
    "names(partitioned_df)\n",
    "class(partitioned_df$normal)\n",
    "partitioned_df$normal %>% names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this function is a **list** with an item for each toy dataset. Each item contains, in turn, both the *full*  dataset and a partition of it in two subsets: the *training* one (*_train*) and the *testing* one (*_test*). We'll see in the next lessons why we need to split a dataset in two partitions. \n",
    "\n",
    "For now, note that the $\\texttt{get}$*_*$\\texttt{partitioned}$*_*$\\texttt{df}()$: provides, for both partitions, the input dataframe (*x_*) and the output dataframe (*y_*). These dataframes can be directly used with **CARET** training function as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Generalized Linear Model \n",
       "\n",
       "80 samples\n",
       " 2 predictor\n",
       " 2 classes: 'class_1', 'class_2' \n",
       "\n",
       "No pre-processing\n",
       "Resampling: Bootstrapped (25 reps) \n",
       "Summary of sample sizes: 80, 80, 80, 80, 80, 80, ... \n",
       "Resampling results:\n",
       "\n",
       "  Accuracy  Kappa\n",
       "  1         1    \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train(x = partitioned_df$normal$x_train,\n",
    "      y = partitioned_df$normal$y_train$class, \n",
    "      method = \"glm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that here we used method **glm** (*Generalized Linear Model*) since we are trying to fit a classification problem while the *simpler* **lm** method only works with regression problems (try to replace **glm** with **lm** in the cell above...)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}