Day27- 實作樹型模型_作業.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 作業目的: 實作樹型模型\n",
    "\n",
    "在本次課程中實作了以Entropy計算訊息增益的決策樹模型，而計算訊息增益的方法除了Entropy只外還有Gini。因此本次作業希望讀者實作以Gini計算\n",
    "\n",
    "訊息增益，且基於課程的決策樹模型建構隨機森林模型。\n",
    "\n",
    "在作業資料夾中的`decision_tree_functions.py`檔案有在作業中實作的所有函式，在實作作業中可以充分利用已經寫好的函式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q1: 使用Gini計算訊息增益\n",
    "\n",
    "$$\n",
    "Gini = \\sum_{i=1}^cp(i)(1-p(i)) = 1 - \\sum_{i=1}^cp(i)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from decision_tree_functions import decision_tree, train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "    color  diameter  label\n",
       "0   Green       3.1  Apple\n",
       "1     Red       3.2  Apple\n",
       "2     Red       1.2  Grape\n",
       "3     Red       1.0  Grape\n",
       "4  Yellow       3.3  Lemon"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>color</th>\n      <th>diameter</th>\n      <th>label</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Green</td>\n      <td>3.1</td>\n      <td>Apple</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Red</td>\n      <td>3.2</td>\n      <td>Apple</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Red</td>\n      <td>1.2</td>\n      <td>Grape</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Red</td>\n      <td>1.0</td>\n      <td>Grape</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Yellow</td>\n      <td>3.3</td>\n      <td>Lemon</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "# 使用與課程中相同的假資料\n",
    "\n",
    "training_data = [\n",
    "    ['Green', 3.1, 'Apple'],\n",
    "    ['Red', 3.2, 'Apple'],\n",
    "    ['Red', 1.2, 'Grape'],\n",
    "    ['Red', 1, 'Grape'],\n",
    "    ['Yellow', 3.3, 'Lemon'],\n",
    "    ['Yellow', 3.1, 'Lemon'],\n",
    "    ['Green', 3, 'Apple'],\n",
    "    ['Red', 1.1, 'Grape'],\n",
    "    ['Yellow', 3, 'Lemon'],\n",
    "    ['Red', 1.2, 'Grape'],\n",
    "]\n",
    "\n",
    "header = [\"color\", \"diameter\", \"label\"]\n",
    "\n",
    "df = pd.DataFrame(data=training_data, columns=header)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Gini impurity\n",
    "def calculate_gini(data):\n",
    "    \n",
    "    #取的資料的label訊息\n",
    "    ###<your code>###\n",
    "    label_column = data[:, -1]\n",
    "    #取得所有輸入資料的獨立類別與其個數\n",
    "    ###<your code>###\n",
    "    _, counts = np.unique(label_column, return_counts=True)\n",
    "    #計算機率\n",
    "    ###<your code>###\n",
    "    probabilities = counts / counts.sum()\n",
    "    #計算gini impurity\n",
    "    ###<your code>###\n",
    "    gini=1.00 -np.sum(np.square(probabilities))\n",
    "\n",
    "    return gini"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'diameter <= 1.2': ['Grape', {'color = Yellow': ['Lemon', 'Apple']}]}"
      ]
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "source": [
    "#分割資料集\n",
    "###<your code>###\n",
    "train_df, test_df = train_test_split(df, 0.2)\n",
    "#以Gini inpurity作為metric_function訓練決策樹\n",
    "tree = decision_tree(calculate_gini, 'classification', 0, min_samples=2, max_depth=5) ###<your code>###\n",
    "tree.fit(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'Lemon'"
      ]
     },
     "metadata": {},
     "execution_count": 14
    }
   ],
   "source": [
    "# 以建構好的樹進行預測\n",
    "sample = test_df.iloc[0]\n",
    "###<your code>###\n",
    "tree.pred(sample,tree.sub_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "color       Yellow\n",
       "diameter         3\n",
       "label        Lemon\n",
       "Name: 8, dtype: object"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "source": [
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q2: 實作隨機森林\n",
    "利用決策樹來實作隨機森林模型，讀者可參考隨機森林課程講義。\n",
    "\n",
    "此份作業只要求讀者實作隨機sample訓練資料，而隨機sample特徵進行訓練的部分，讀者可以參考`decision_tree_functions.py`中的`get_potential_splits`與`decision_tree`部分(新增參數`random_features`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class random_forest():\n",
    "    '''Random forest model\n",
    "    Parameters\n",
    "    ----------\n",
    "    n_boostrap: int\n",
    "        number of samples to sample to train indivisual decision tree\n",
    "    n_tree: int\n",
    "        number of trees to form a forest\n",
    "    '''\n",
    "    \n",
    "    def __init__(self, n_bootstrap, n_trees, task_type, min_samples, max_depth, metric_function, n_features=None):\n",
    "        self.n_bootstrap = n_bootstrap\n",
    "        self.n_trees = n_trees\n",
    "        self.task_type = task_type\n",
    "        self.min_samples = min_samples\n",
    "        self.max_depth = max_depth\n",
    "        self.metric_function = metric_function\n",
    "        self.n_features = n_features\n",
    "    \n",
    "    def bootstrapping(self, train_df, n_bootstrap):\n",
    "        #sample data to be used to train individual tree\n",
    "        ###<your code>###\n",
    "        bootstrap_idxs = np.random.randint(0, high=len(train_df), size=n_bootstrap)\n",
    "        df_bootstrapped = train_df.iloc[bootstrap_idxs]\n",
    "        #avoid pick the samples with all the same label\n",
    "        ###<your code>###\n",
    "        if len(np.unique(df_bootstrapped['label'])) == 1:\n",
    "            return self.bootstrapping(train_df, n_bootstrap)\n",
    "\n",
    "        return df_bootstrapped\n",
    "    \n",
    "    def fit(self, train_df):\n",
    "        \n",
    "        self.forest = []\n",
    "        \n",
    "        ###<your code>###\n",
    "        for i in range(self.n_trees):\n",
    "            df_bootstrapped = self.bootstrapping(train_df, self.n_bootstrap)\n",
    "            tree = decision_tree(self.metric_function,\n",
    "                                 task_type=self.task_type,\n",
    "                                 counter=0,\n",
    "                                 min_samples=self.min_samples,\n",
    "                                 max_depth=self.max_depth,\n",
    "                                 random_features=self.n_features)\n",
    "            tree.fit(df_bootstrapped)\n",
    "            self.forest.append(tree)\n",
    "\n",
    "        return self.forest\n",
    "    \n",
    "    def pred(self, test_df):\n",
    "        df_predictions = {}\n",
    "        \n",
    "        ###<your code>###\n",
    "        for i in range(self.n_trees):\n",
    "            col_name = f\"tree_{i}\"\n",
    "            df_predictions[col_name] = test_df.apply(self.forest[i].pred, args=(self.forest[i].sub_tree,), axis=1)\n",
    "\n",
    "        df_predictions = pd.DataFrame(df_predictions)\n",
    "        \n",
    "        #majority voting\n",
    "        ###<your code>###\n",
    "        random_forest_predictions = df_predictions.mode(axis=1)[0]\n",
    "\n",
    "        return random_forest_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[<decision_tree_functions.decision_tree at 0x29377543608>,\n",
       " <decision_tree_functions.decision_tree at 0x293775dcfc8>,\n",
       " <decision_tree_functions.decision_tree at 0x293775431c8>]"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "source": [
    "train_df, test_df = train_test_split(df, 0.2)\n",
    "\n",
    "#建立隨機森林模型\n",
    "###<your code>###\n",
    "forest = random_forest(n_bootstrap=25,\n",
    "                       n_trees=3,\n",
    "                       task_type='classification',\n",
    "                       min_samples=3,\n",
    "                       max_depth=4,\n",
    "                       metric_function=calculate_gini,\n",
    "                       n_features=None)\n",
    "forest.fit(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "4    Lemon\n",
       "9    Grape\n",
       "Name: 0, dtype: object"
      ]
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "source": [
    "forest.pred(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "    color  diameter  label\n",
       "4  Yellow       3.3  Lemon\n",
       "9     Red       1.2  Grape"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>color</th>\n      <th>diameter</th>\n      <th>label</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>4</th>\n      <td>Yellow</td>\n      <td>3.3</td>\n      <td>Lemon</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Red</td>\n      <td>1.2</td>\n      <td>Grape</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 20
    }
   ],
   "source": [
    "test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.6 64-bit (conda)",
   "metadata": {
    "interpreter": {
     "hash": "7b4b4feff2f24a0f0a34464dbe537a36fda679851528fb8735cb41fa49dffb2d"
    }
   }
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}