Predicting Wine Types with Decision Trees & Random Forest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Wine Types with Decision Trees & Random Forest\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This notebook is a simple demonstration of how to use scikit-learn to build a Decision Tree model and Random Forest model for classification. It uses a dataset of 178 wines and their various attributes. There are three different classes of wine in the data and the goal is to predict the wine class based upon the attributes.\n",
    "\n",
    "## The Data\n",
    "\n",
    "The data has been taken from [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml) and can be found [here](https://archive.ics.uci.edu/ml/datasets/Wine). \n",
    "\n",
    "Information on the data can be found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.names). The main thing we are going to focus on our the attribute names as the raw data doesn't have a header. The first column is the class, which is what we wish to predict, and the rest of the attributes we will use as features:\n",
    "1. Alcohol\n",
    "2. Malic acid\n",
    "3. Ash\n",
    "4. Alcalinity of ash  \n",
    "5. Magnesium\n",
    "6. Total phenols\n",
    "7. Flavanoids\n",
    "8. Nonflavanoid phenols\n",
    "9. Proanthocyanins\n",
    "10. Color intensity\n",
    "11. Hue\n",
    "12. OD280/OD315 of diluted wines\n",
    "13. Proline   \n",
    "\n",
    "We import python libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We read in the data we've saved, passing the column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wine = pd.read_csv(\"data/wine.csv\", names=[\"class\", \"alcohol\", \"malic_acid\", \"ash\", \"alcalinity_of_ash\", \"magnesium\", \"total_phenols\",\n",
    "                                          \"flavanoids\", \"nonflavanoid_phenols\", \"proanthocyanins\", \"color_intensity\", \"hue\", \n",
    "                                           \"od280_od315_of_diluted_wines\", \"proline\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check out the first few rows of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280_od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   class  alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  \\\n",
       "0      1    14.23        1.71  2.43               15.6        127   \n",
       "1      1    13.20        1.78  2.14               11.2        100   \n",
       "2      1    13.16        2.36  2.67               18.6        101   \n",
       "3      1    14.37        1.95  2.50               16.8        113   \n",
       "4      1    13.24        2.59  2.87               21.0        118   \n",
       "\n",
       "   total_phenols  flavanoids  nonflavanoid_phenols  proanthocyanins  \\\n",
       "0           2.80        3.06                  0.28             2.29   \n",
       "1           2.65        2.76                  0.26             1.28   \n",
       "2           2.80        3.24                  0.30             2.81   \n",
       "3           3.85        3.49                  0.24             2.18   \n",
       "4           2.80        2.69                  0.39             1.82   \n",
       "\n",
       "   color_intensity   hue  od280_od315_of_diluted_wines  proline  \n",
       "0             5.64  1.04                          3.92     1065  \n",
       "1             4.38  1.05                          3.40     1050  \n",
       "2             5.68  1.03                          3.17     1185  \n",
       "3             7.80  0.86                          3.45     1480  \n",
       "4             4.32  1.04                          2.93      735  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train & Test Data\n",
    "\n",
    "The purpose of splitting the data is to be able to assess the quality of a predictive model when it is used on unseen data. When training, you will try to build a model that fits to the data as closely as possible, to be able to most accurately make a prediction. However, without a test set you run the risk of overfitting - the model works very well for the data it has seen but not for new data.\n",
    "\n",
    "The split ratio is often debated and in practice you might split your data into three sets: train, validation and test. You would use the training data to understand which classifier you wish to use; the validation set to test on whilst tweaking parameters; and the test set to get an understanding of how your final model would work in practice. Furthermore, there are techniques such as K-Fold cross validation that also help to reduce bias.\n",
    "\n",
    "For the purpose of this demonstration, we will only be randomly splitting our data into test and train, with a 80/20 split.\n",
    "\n",
    "We import the required library from scikit-learn, [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wish for all features to be used for training, therefore we are taking all columns except \"class\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X = wine.drop([\"class\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column \"class\" is our target variable, we set y as this column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = wine[\"class\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the *train_test_split* function to create the appropriate train and test data for our features (\"X_train\" and \"X_test\" respectively) and target data (\"Y_train\" and \"Y_test\"). We are specifying our test data to be 20% of the total data. We are also providing a seed to be able to reproduce this split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the number of examples we have in each of our train and test data sets using \"shape\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(142, 13)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(36, 13)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standardisation\n",
    "\n",
    "All features are numeric so we do not need to worry about converting categorical data with techniques such as one-hot encoding. However, we will demonstrate how to standardise our data. Standardisation rescales our attributes so they have a mean of 0 and standard deviation of 1. It assumes that the distribution is Gaussian (it works better if it is), alternatively normalisation can be used to rescale between the range of 0 and 1\n",
    "\n",
    "We use scikit-learn's [StandardScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create the scaler, leaving parameters as default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We fit the scaler passing the training data but also request it transforms the data and returns it to a variable named \"train_scaled\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_scaled = scaler.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then transform our test data with the same fitted scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Trees & Random Forest\n",
    "\n",
    "Decision trees learn how to best split the dataset into separate branches, allowing it to learn non-linear relationships. \n",
    "\n",
    "Random forests (RF) combine predictions from many individual decision trees. As they use a collection of results to make a final decision, they are referred to as 'Ensemble techniques'.\n",
    "\n",
    "We are using scikit-learn's [Decision Tree Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html) and [Random Forest Classifier](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an Decision Tree model & a Random Forest Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tree_model = DecisionTreeClassifier()\n",
    "rf_model = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train them with our scaled training data and target values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=None, verbose=0,\n",
       "            warm_start=False)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_model.fit(train_scaled, y_train)\n",
    "rf_model.fit(train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "\n",
    "We wish to understand how good our model is; one simple metric to use for evaluation is accuracy. This will return the percentage of correct predictions.\n",
    "\n",
    "We import [scikit-learn's accuracy_score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We calculate accuracy for our training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree training accuracy 1.0\n",
      "Random Forest training accuracy 1.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Decision Tree training accuracy\",accuracy_score(y_train, tree_model.predict(train_scaled)))\n",
    "print(\"Random Forest training accuracy\",accuracy_score(y_train, rf_model.predict(train_scaled)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both result in training accuracy of 100%! More importantly, we should check if we find good results on unseen data (to ensure we haven't overfit). We calculate the accuracy for our test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree test accuracy 0.8888888888888888\n",
      "Random Forest test accuracy 0.9722222222222222\n"
     ]
    }
   ],
   "source": [
    "print(\"Decision Tree test accuracy\",accuracy_score(y_test, tree_model.predict(test_scaled)))\n",
    "print(\"Random Forest test accuracy\",accuracy_score(y_test, rf_model.predict(test_scaled)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it looks like our Decision Tree overfit resulting in a lower accuracy on our test data. However, our Random Forest model looks much better with an accuracy of 97.2%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Trees & Random Forest Parameters\n",
    "\n",
    "More information on tree algorithms can be found in the scikit-learn documentation [here](http://scikit-learn.org/stable/modules/tree.html) and ensembles [here](http://scikit-learn.org/stable/modules/ensemble.html)\n",
    "\n",
    "There are a number of parameters that can be tuned that should be explored when trying to improve Decision Trees and Random Forest models. A common approach is to test many different paramters, building multiple models and testing their accuracy to find the best combination.\n",
    "\n",
    "### Decision Trees\n",
    "For Decision Trees, the [scikit-learn documentation](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier) provides parameters that can be passed by the user; changing these are likely to have an impact on the performance of the model. \n",
    "\n",
    "Here is high-level information on the parameters, the documentation has more details:\n",
    "- criterion : default=”gini”\n",
    "    - The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain.\n",
    "\n",
    "- splitter : default=”best”\n",
    "    - The strategy used to choose the split at each node. Supported strategies are “best” to choose the best split and “random” to choose the best random split.\n",
    "\n",
    "- max_depth : default=None\n",
    "    - The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.\n",
    "\n",
    "- min_samples_split : default=2\n",
    "    - The minimum number of samples required to split an internal node:\n",
    "\n",
    "- min_samples_leaf : default=1\n",
    "    - The minimum number of samples required to be at a leaf node:\n",
    "\n",
    "- min_weight_fraction_leaf : default=0.\n",
    "    - The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.\n",
    "\n",
    "- max_features : default=None\n",
    "    - The number of features to consider when looking for the best split. None, means take all features\n",
    "\n",
    "- random_state : default=None\n",
    "    - If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.\n",
    "\n",
    "- max_leaf_nodes : default=None\n",
    "    - Grow a tree with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.\n",
    "\n",
    "- min_impurity_decrease : default=0.\n",
    "    - A node will be split if this split induces a decrease of the impurity greater than or equal to this value.\n",
    "\n",
    "- class_weight : default=None\n",
    "    - Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.\n",
    "\n",
    "- presort : default=False\n",
    "    - Whether to presort the data to speed up the finding of best splits in fitting. For the default settings of a decision tree on large datasets, setting this to true may slow down the training process. When using either a smaller dataset or a restricted depth, this may speed up the training.\n",
    "\n",
    "### Random Forests\n",
    "Similarly, for Random Forests, the [scikit-learn documentation](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) provides parameters that can be passed by the user; changing these are likely to have an impact on the performance of the model. \n",
    "\n",
    "Here is high-level information on the parameters, the documentation has more details:\n",
    "- n_estimators : default=10\n",
    "    - The number of trees in the forest.\n",
    "\n",
    "- criterion : default=”gini”\n",
    "    - The function to measure the quality of a split. Supported criteria are “gini” for the Gini impurity and “entropy” for the information gain. Note: this parameter is tree-specific.\n",
    "\n",
    "- max_features : default=”auto”\n",
    "    - The number of features to consider when looking for the best split:\n",
    "\n",
    "- max_depth : default=None\n",
    "    - The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.\n",
    "\n",
    "- min_samples_split : default=2\n",
    "    - The minimum number of samples required to split an internal node:\n",
    "\n",
    "- min_samples_leaf : default=1\n",
    "    - The minimum number of samples required to be at a leaf node:\n",
    "\n",
    "- min_weight_fraction_leaf : default=0.\n",
    "    - The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.\n",
    "\n",
    "- max_leaf_nodes : default=None\n",
    "    - Grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.\n",
    "\n",
    "- min_impurity_decrease : default=0.\n",
    "    - A node will be split if this split induces a decrease of the impurity greater than or equal to this value.\n",
    "\n",
    "- bootstrap : default=True\n",
    "    - Whether bootstrap samples are used when building trees.\n",
    "\n",
    "- oob_score : default=False\n",
    "    - Whether to use out-of-bag samples to estimate the generalization accuracy.\n",
    "\n",
    "- n_jobs : default=1\n",
    "    - The number of jobs to run in parallel for both fit and predict. If -1, then the number of jobs is set to the number of cores.\n",
    "\n",
    "- random_state : default=None\n",
    "    - If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.\n",
    "\n",
    "- verbose : default=0\n",
    "    - Controls the verbosity of the tree building process.\n",
    "\n",
    "- warm_start : default=False\n",
    "    - When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest.\n",
    "\n",
    "- class_weight : default=None\n",
    "    - “balanced_subsample” or None, optional. Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.\n",
    "    \n",
    "### Grid Search\n",
    "\n",
    "To search for the best hyper-parameters for your algorithm and data, grid search cross validation is commonly used. The [scikit-learn documentation](http://scikit-learn.org/stable/modules/grid_search.html) provides more thorough information on how to use this. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Citation\n",
    "\n",
    "Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}