Predicting Yacht Resistance with K Nearest Neighbors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Yacht Resistance with K Nearest Neighbors\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This notebook is a simple demonstration of how to use scikit-learn to build a K-nearest neighbor (KNN) model for regression. It uses a dataset of 308 experiments and their various attributes. The goal is to predict the residuary resistance per unit weight of displacement based upon the attributes.\n",
    "\n",
    "## The Data\n",
    "\n",
    "The data has been taken from [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml) and the raw data and information can be found [here](https://archive.ics.uci.edu/ml/datasets/Yacht+Hydrodynamics). \n",
    "\n",
    "The columns are as follow:\n",
    "\n",
    "1. Longitudinal position of the center of buoyancy, adimensional.\n",
    "2. Prismatic coefficient, adimensional.\n",
    "3. Length-displacement ratio, adimensional.\n",
    "4. Beam-draught ratio, adimensional.\n",
    "5. Length-beam ratio, adimensional.\n",
    "6. Froude number, adimensional.\n",
    "7. Residuary resistance per unit weight of displacement, adimensional. \n",
    "\n",
    "Where column 7 is the target variable we are looking to predict.\n",
    "\n",
    "We import python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We read in the data we've saved, passing the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "yacht = pd.read_csv(\"data/yacht_hydrodynamics.csv\", names=[\"longitudinal_pos\", \"presmatic_coef\", \"length_disp\", \"beam-draught_rt\", \n",
    "                                                           \"length-beam_rt\", \"froude_num\", \"resid_resist\"], sep=\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check out the first few rows of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitudinal_pos</th>\n",
       "      <th>presmatic_coef</th>\n",
       "      <th>length_disp</th>\n",
       "      <th>beam-draught_rt</th>\n",
       "      <th>length-beam_rt</th>\n",
       "      <th>froude_num</th>\n",
       "      <th>resid_resist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.3</td>\n",
       "      <td>0.568</td>\n",
       "      <td>4.78</td>\n",
       "      <td>3.99</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.125</td>\n",
       "      <td>0.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-2.3</td>\n",
       "      <td>0.568</td>\n",
       "      <td>4.78</td>\n",
       "      <td>3.99</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-2.3</td>\n",
       "      <td>0.568</td>\n",
       "      <td>4.78</td>\n",
       "      <td>3.99</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.175</td>\n",
       "      <td>0.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-2.3</td>\n",
       "      <td>0.568</td>\n",
       "      <td>4.78</td>\n",
       "      <td>3.99</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.200</td>\n",
       "      <td>0.78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-2.3</td>\n",
       "      <td>0.568</td>\n",
       "      <td>4.78</td>\n",
       "      <td>3.99</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.225</td>\n",
       "      <td>1.18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitudinal_pos  presmatic_coef  length_disp  beam-draught_rt  \\\n",
       "0              -2.3           0.568         4.78             3.99   \n",
       "1              -2.3           0.568         4.78             3.99   \n",
       "2              -2.3           0.568         4.78             3.99   \n",
       "3              -2.3           0.568         4.78             3.99   \n",
       "4              -2.3           0.568         4.78             3.99   \n",
       "\n",
       "   length-beam_rt  froude_num  resid_resist  \n",
       "0            3.17       0.125          0.11  \n",
       "1            3.17       0.150          0.27  \n",
       "2            3.17       0.175          0.47  \n",
       "3            3.17       0.200          0.78  \n",
       "4            3.17       0.225          1.18  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yacht.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can quickly check if we have any null values in our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yacht.isnull().values.any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do! Let's use the \"describe\" method to find them, amongst other interesting information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitudinal_pos</th>\n",
       "      <th>presmatic_coef</th>\n",
       "      <th>length_disp</th>\n",
       "      <th>beam-draught_rt</th>\n",
       "      <th>length-beam_rt</th>\n",
       "      <th>froude_num</th>\n",
       "      <th>resid_resist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>308.000000</td>\n",
       "      <td>252.000000</td>\n",
       "      <td>308.000000</td>\n",
       "      <td>308.000000</td>\n",
       "      <td>308.000000</td>\n",
       "      <td>308.000000</td>\n",
       "      <td>308.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>-2.381818</td>\n",
       "      <td>0.563944</td>\n",
       "      <td>4.008182</td>\n",
       "      <td>4.096364</td>\n",
       "      <td>3.341364</td>\n",
       "      <td>0.824318</td>\n",
       "      <td>8.476461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.513219</td>\n",
       "      <td>0.022947</td>\n",
       "      <td>1.643974</td>\n",
       "      <td>0.653655</td>\n",
       "      <td>0.391571</td>\n",
       "      <td>1.146200</td>\n",
       "      <td>14.052367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-5.000000</td>\n",
       "      <td>0.530000</td>\n",
       "      <td>0.530000</td>\n",
       "      <td>2.810000</td>\n",
       "      <td>2.730000</td>\n",
       "      <td>0.125000</td>\n",
       "      <td>0.010000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-2.400000</td>\n",
       "      <td>0.546000</td>\n",
       "      <td>4.340000</td>\n",
       "      <td>3.750000</td>\n",
       "      <td>3.150000</td>\n",
       "      <td>0.225000</td>\n",
       "      <td>0.367500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-2.300000</td>\n",
       "      <td>0.565000</td>\n",
       "      <td>4.780000</td>\n",
       "      <td>3.990000</td>\n",
       "      <td>3.170000</td>\n",
       "      <td>0.325000</td>\n",
       "      <td>1.790000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>-2.300000</td>\n",
       "      <td>0.574000</td>\n",
       "      <td>4.780000</td>\n",
       "      <td>4.770000</td>\n",
       "      <td>3.530000</td>\n",
       "      <td>0.425000</td>\n",
       "      <td>8.092500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>5.140000</td>\n",
       "      <td>5.350000</td>\n",
       "      <td>4.240000</td>\n",
       "      <td>3.510000</td>\n",
       "      <td>62.420000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitudinal_pos  presmatic_coef  length_disp  beam-draught_rt  \\\n",
       "count        308.000000      252.000000   308.000000       308.000000   \n",
       "mean          -2.381818        0.563944     4.008182         4.096364   \n",
       "std            1.513219        0.022947     1.643974         0.653655   \n",
       "min           -5.000000        0.530000     0.530000         2.810000   \n",
       "25%           -2.400000        0.546000     4.340000         3.750000   \n",
       "50%           -2.300000        0.565000     4.780000         3.990000   \n",
       "75%           -2.300000        0.574000     4.780000         4.770000   \n",
       "max            0.000000        0.600000     5.140000         5.350000   \n",
       "\n",
       "       length-beam_rt  froude_num  resid_resist  \n",
       "count      308.000000  308.000000    308.000000  \n",
       "mean         3.341364    0.824318      8.476461  \n",
       "std          0.391571    1.146200     14.052367  \n",
       "min          2.730000    0.125000      0.010000  \n",
       "25%          3.150000    0.225000      0.367500  \n",
       "50%          3.170000    0.325000      1.790000  \n",
       "75%          3.530000    0.425000      8.092500  \n",
       "max          4.240000    3.510000     62.420000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yacht.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So... the column *presmatic_coef* has 56 missing values... we can deal with this in a few different ways. The simpliest solution is to remove them, though we lose many examples in doing so. Alternatively, we could impute the values, replacing the NaN values with an average (mean or median). For the purpose of this simple notebook, we will simply remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "yacht = yacht.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train & Test Data\n",
    "\n",
    "The purpose of splitting the data is to be able to assess the quality of a predictive model when it is used on unseen data. When training, you will try to build a model that fits to the data as closely as possible, to be able to most accurately make a prediction. However, without a test set you run the risk of overfitting - the model works very well for the data it has seen but not for new data.\n",
    "\n",
    "The split ratio is often debated and in practice you might split your data into three sets: train, validation and test. You would use the training data to understand which classifier you wish to use; the validation set to test on whilst tweaking parameters; and the test set to get an understanding of how your final model would work in practice. Furthermore, there are techniques such as K-Fold cross validation that also help to reduce bias.\n",
    "\n",
    "For the purpose of this demonstration, we will only be randomly splitting our data into test and train, with a 80/20 split.\n",
    "\n",
    "We import the required library from scikit-learn, [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wish for all features to be used for training, therefore we are taking all columns except \"class\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = yacht.drop([\"resid_resist\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column \"class\" is our target variable, we set y as this column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = yacht[\"resid_resist\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the *train_test_split* function to create the appropriate train and test data for our features (\"X_train\" and \"X_test\" respectively) and target data (\"Y_train\" and \"Y_test\"). We are specifying our test data to be 20% of the total data. We are also providing a seed to be able to reproduce this split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the number of examples we have in each of our train and test data sets using \"shape\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(201, 6)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(51, 6)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardisation\n",
    "\n",
    "All features are numeric so we do not need to worry about converting categorical data with techniques such as one-hot encoding. However, we will demonstrate how to standardise our data. Standardisation rescales our attributes so they have a mean of 0 and standard deviation of 1. It assumes that the distribution is Gaussian (it works better if it is), alternatively normalisation can be used to rescale between the range of 0 and 1\n",
    "\n",
    "We use scikit-learn's [StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the scaler, leaving parameters as default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit the scaler passing the training data but also request it transforms the data and returns it to a variable named \"train_scaled\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_scaled = scaler.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then transform our test data with the same fitted scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K Nearest Neighbors\n",
    "\n",
    "K-Nearest Neighbor (KNN) makes a prediction for a new observation by searching for the most similar training observations and pooling their values.\n",
    "\n",
    "We are using scikit-learn's [K Neighbors Regressor](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an KNN model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = KNeighborsRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train it with our scaled training data and target values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "          metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
       "          weights='uniform')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "\n",
    "We wish to understand how good our model is; there are a few different metrics we can use. We will evaluate mean squared error (MSE) and mean absolute error (MAE)\n",
    "\n",
    "We import [scikit-learn's mean squared error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error) and [sckit-learn's mean absolute error](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import mean_absolute_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We calculate the errors for our training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mse = mean_squared_error(y_train, model.predict(train_scaled))\n",
    "mae = mean_absolute_error(y_train, model.predict(train_scaled))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import sqrt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mse =  83.07649245771145  & mae =  4.645452736318409  & rmse =  9.114630681366714\n"
     ]
    }
   ],
   "source": [
    "print(\"mse = \",mse,\" & mae = \",mae,\" & rmse = \", sqrt(mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easier metric to understand is the mean absolute error, this means that on average our prediction was 4.6 away from the true prediction. Mean squared error, and consequently root mean squared error (RMSE), results in predictions further and further from the true value are punished more.\n",
    "\n",
    "We can calculate the same on the test data to understand how we the model is generalised."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mse =  30.68350007843137  & mae =  3.08164705882353  & rmse =  5.539268911908084\n"
     ]
    }
   ],
   "source": [
    "test_mse = mean_squared_error(y_test, model.predict(test_scaled))\n",
    "test_mae = mean_absolute_error(y_test, model.predict(test_scaled))\n",
    "print(\"mse = \",test_mse,\" & mae = \",test_mae,\" & rmse = \", sqrt(test_mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are actually seeing better results on our test data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K Nearest Neighbors Parameters\n",
    "\n",
    "More information on Nearest Neighbors can be found in the scikit-learn documentation [here](http://scikit-learn.org/stable/modules/neighbors.html)\n",
    "\n",
    "There are a number of parameters that can be tuned that should be explored when trying to improve K Nearest Neighbor models. A common approach is to test many different parameters, building multiple models and testing their accuracy to find the best combination.\n",
    "\n",
    "### Parameters\n",
    "For KNN, the [scikit-learn documentation](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsRegressor.html) provides parameters that can be passed by the user; changing these are likely to have an impact on the performance of the model. \n",
    "\n",
    "Here is high-level information on the parameters, the documentation has more details:\n",
    "- n_neighbors : default = 5\n",
    "    - Number of neighbors to use by default for kneighbors queries.\n",
    "\n",
    "- weights : default='uniform'\n",
    "    - weight function used in prediction\n",
    "\n",
    "- algorithm : default=‘auto’\n",
    "    - Algorithm used to compute the nearest neighbors:\n",
    "\n",
    "- leaf_size : default = 30\n",
    "    - Leaf size passed to BallTree or KDTree. This can affect the speed of the construction and query, as well as the memory required to store the tree. The optimal value depends on the nature of the problem.\n",
    "\n",
    "- p : default = 2\n",
    "    - Power parameter for the Minkowski metric. When p = 1, this is equivalent to using manhattan_distance (l1), and euclidean_distance (l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.\n",
    "\n",
    "- metric : default ‘minkowski’\n",
    "    - The distance metric to use for the tree. The default metric is minkowski, and with p=2 is equivalent to the standard Euclidean metric. See the documentation of the DistanceMetric class for a list of available metrics.\n",
    "\n",
    "- metric_params : default = None\n",
    "    - Additional keyword arguments for the metric function.\n",
    "\n",
    "- n_jobs : default = 1\n",
    "    - The number of parallel jobs to run for neighbors search. If -1, then the number of jobs is set to the number of CPU cores. Doesn’t affect fit method.\n",
    "\n",
    "\n",
    "### Grid Search\n",
    "\n",
    "To search for the best hyper-parameters for your algorithm and data, grid search cross validation is commonly used. The [scikit-learn documentation](http://scikit-learn.org/stable/modules/grid_search.html) provides more thorough information on how to use this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Citation\n",
    "\n",
    "Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}