Day_049_Blending_HW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 作業 : (Kaggle)鐵達尼生存預測\n",
    "***\n",
    "- 分數以網站評分結果為準, 請同學實際將提交檔(*.csv)上傳試試看  \n",
    "https://www.kaggle.com/c/titanic/submit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [作業目標]\n",
    "- 試著模仿範例寫法, 在鐵達尼生存預測中, 觀察觀查混合泛化 (Blending) 的寫法與效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [作業重點]\n",
    "- 觀察混合泛化的準確度 (In[14]), 是否比單一模型準確度為高 (In[11~13])  \n",
    "- 除了我們的權重, 同學也可以試著自行調整權重 (注意:權重和=1), 看看有什麼影響\n",
    "- Hint : 除了權重, 分類預測的調整, 還可以調整什麼地方?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   Pclass                                               Name     Sex   Age  \\\n0       3                            Braund, Mr. Owen Harris    male  22.0   \n1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0   \n2       3                             Heikkinen, Miss. Laina  female  26.0   \n3       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0   \n4       3                           Allen, Mr. William Henry    male  35.0   \n\n   SibSp  Parch            Ticket     Fare Cabin Embarked  \n0      1      0         A/5 21171   7.2500   NaN        S  \n1      1      0          PC 17599  71.2833   C85        C  \n2      0      0  STON/O2. 3101282   7.9250   NaN        S  \n3      1      0            113803  53.1000  C123        S  \n4      0      0            373450   8.0500   NaN        S  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Pclass</th>\n      <th>Name</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Ticket</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>3</td>\n      <td>Braund, Mr. Owen Harris</td>\n      <td>male</td>\n      <td>22.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>A/5 21171</td>\n      <td>7.2500</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n      <td>female</td>\n      <td>38.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>PC 17599</td>\n      <td>71.2833</td>\n      <td>C85</td>\n      <td>C</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>Heikkinen, Miss. Laina</td>\n      <td>female</td>\n      <td>26.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>STON/O2. 3101282</td>\n      <td>7.9250</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n      <td>female</td>\n      <td>35.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>113803</td>\n      <td>53.1000</td>\n      <td>C123</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3</td>\n      <td>Allen, Mr. William Henry</td>\n      <td>male</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>373450</td>\n      <td>8.0500</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "source": [
    "# 做完特徵工程前的所有準備 (與前範例相同)\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import copy, time\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from IPython.display import display\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "data_path = 'C:/Users/user/OneDrive/cupoy/Machine Learning/教材/data/'\n",
    "df_train = pd.read_csv(data_path + 'titanic_train.csv')\n",
    "df_test = pd.read_csv(data_path + 'titanic_test.csv')\n",
    "\n",
    "train_Y = df_train['Survived']\n",
    "ids = df_test['PassengerId']\n",
    "df_train = df_train.drop(['PassengerId', 'Survived'] , axis=1)\n",
    "df_test = df_test.drop(['PassengerId'] , axis=1)\n",
    "df = pd.concat([df_train,df_test])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "          Missing Ratio\nCabin         77.463713\nAge           20.091673\nEmbarked       0.152788\nFare           0.076394",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Missing Ratio</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>Cabin</th>\n      <td>77.463713</td>\n    </tr>\n    <tr>\n      <th>Age</th>\n      <td>20.091673</td>\n    </tr>\n    <tr>\n      <th>Embarked</th>\n      <td>0.152788</td>\n    </tr>\n    <tr>\n      <th>Fare</th>\n      <td>0.076394</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {}
    }
   ],
   "source": [
    "# 檢查 DataFrame 空缺值的狀態\n",
    "def na_check(df_data):\n",
    "    data_na = (df_data.isnull().sum() / len(df_data)) * 100\n",
    "    data_na = data_na.drop(data_na[data_na == 0].index).sort_values(ascending=False)\n",
    "    missing_data = pd.DataFrame({'Missing Ratio' :data_na})\n",
    "    display(missing_data.head(10))\n",
    "na_check(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 以下 In[3]~In[10] 只是鐵達尼預測中的一組特徵工程, 並以此組特徵工程跑參數, 若更換其他特徵工程, In[10]的參數需要重新跑\n",
    "# Sex : 直接轉男 0 女 1\n",
    "df[\"Sex\"] = df[\"Sex\"].map({\"male\": 0, \"female\":1})\n",
    "# Fare : 用 log 去偏態, 0 則直接取 0\n",
    "df[\"Fare\"] = df[\"Fare\"].map(lambda i: np.log(i) if i > 0 else 0)\n",
    "# Age : 缺值用中位數補\n",
    "df[\"Age\"] = df[\"Age\"].fillna(df['Age'].median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Title 的 特徵工程 : 將各種頭銜按照類型分類, 最後取 One Hot\n",
    "df_title = [i.split(\",\")[1].split(\".\")[0].strip() for i in df[\"Name\"]]\n",
    "df[\"Title\"] = pd.Series(df_title)\n",
    "df[\"Title\"] = df[\"Title\"].replace(['Lady', 'the Countess','Countess','Capt', 'Col','Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')\n",
    "df[\"Title\"] = df[\"Title\"].map({\"Master\":0, \"Miss\":1, \"Ms\" : 1 , \"Mme\":1, \"Mlle\":1, \"Mrs\":1, \"Mr\":2, \"Rare\":3})\n",
    "df[\"Title\"] = df[\"Title\"].astype(int)\n",
    "df = pd.get_dummies(df, columns = [\"Title\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 新建:家庭大小 (Fsize)特徵, 並依照大小分別建獨立欄位\n",
    "df[\"Fsize\"] = df[\"SibSp\"] + df[\"Parch\"] + 1\n",
    "df['Single'] = df['Fsize'].map(lambda s: 1 if s == 1 else 0)\n",
    "df['SmallF'] = df['Fsize'].map(lambda s: 1 if  s == 2  else 0)\n",
    "df['MedF'] = df['Fsize'].map(lambda s: 1 if 3 <= s <= 4 else 0)\n",
    "df['LargeF'] = df['Fsize'].map(lambda s: 1 if s >= 5 else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ticket : 如果不只是數字-取第一個空白之前的字串(去除'.'與'/'), 如果只是數字-設為'X', 最後再取 One Hot\n",
    "Ticket = []\n",
    "for i in list(df.Ticket):\n",
    "    if not i.isdigit() :\n",
    "        Ticket.append(i.replace(\".\",\"\").replace(\"/\",\"\").strip().split(' ')[0])\n",
    "    else:\n",
    "        Ticket.append(\"X\")        \n",
    "df[\"Ticket\"] = Ticket\n",
    "df = pd.get_dummies(df, columns = [\"Ticket\"], prefix=\"T\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Cabib 依照第一碼分類, 再取 One Hot\n",
    "df[\"Cabin\"] = pd.Series([i[0] if not pd.isnull(i) else 'X' for i in df['Cabin'] ])\n",
    "df = pd.get_dummies(df, columns = [\"Cabin\"], prefix=\"Cabin\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Embarked, Pclass 取 One Hot\n",
    "df = pd.get_dummies(df, columns = [\"Embarked\"], prefix=\"Em\")\n",
    "df[\"Pclass\"] = df[\"Pclass\"].astype(\"category\")\n",
    "df = pd.get_dummies(df, columns = [\"Pclass\"], prefix=\"Pc\")\n",
    "\n",
    "# 捨棄 Name 欄位\n",
    "df.drop(labels = [\"Name\"], axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "Empty DataFrame\nColumns: [Missing Ratio]\nIndex: []",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Missing Ratio</th>\n    </tr>\n  </thead>\n  <tbody>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {}
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   Sex   Age  SibSp  Parch      Fare  Title_0  Title_1  Title_2  Title_3  T_A  \\\n0    0  22.0      1      0  1.981001        0        0        1        0    0   \n1    1  38.0      1      0  4.266662        0        1        0        0    0   \n2    1  26.0      0      0  2.070022        0        1        0        0    0   \n3    1  35.0      1      0  3.972177        0        1        0        0    0   \n4    0  35.0      0      0  2.085672        0        0        1        0    0   \n\n   ...  Cabin_F  Cabin_G  Cabin_T  Cabin_X  Em_C  Em_Q  Em_S  Pc_1  Pc_2  Pc_3  \n0  ...        0        0        0        1     0     0     1     0     0     1  \n1  ...        0        0        0        0     1     0     0     1     0     0  \n2  ...        0        0        0        1     0     0     1     0     0     1  \n3  ...        0        0        0        0     0     0     1     1     0     0  \n4  ...        0        0        0        1     0     0     1     0     0     1  \n\n[5 rows x 61 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n      <th>Title_0</th>\n      <th>Title_1</th>\n      <th>Title_2</th>\n      <th>Title_3</th>\n      <th>T_A</th>\n      <th>...</th>\n      <th>Cabin_F</th>\n      <th>Cabin_G</th>\n      <th>Cabin_T</th>\n      <th>Cabin_X</th>\n      <th>Em_C</th>\n      <th>Em_Q</th>\n      <th>Em_S</th>\n      <th>Pc_1</th>\n      <th>Pc_2</th>\n      <th>Pc_3</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>22.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1.981001</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>38.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>4.266662</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>26.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2.070022</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>1</td>\n      <td>35.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>3.972177</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2.085672</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 61 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "# 確認缺值 與 目前的資料表內容\n",
    "na_check(df)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 將資料最大最小化\n",
    "df = MinMaxScaler().fit_transform(df)\n",
    "\n",
    "# 將前述轉換完畢資料 df , 重新切成 train_X, test_X\n",
    "train_num = train_Y.shape[0]\n",
    "train_X = df[:train_num]\n",
    "test_X = df[train_num:]\n",
    "\n",
    "# 使用三種模型 : 邏輯斯迴歸 / 梯度提升機 / 隨機森林, 參數使用 Random Search 尋找\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import GradientBoostingClassifier, RandomForestClassifier\n",
    "lr = LogisticRegression(tol=0.001, penalty='l2', fit_intercept=True, C=1.0)\n",
    "gdbt = GradientBoostingClassifier(tol=100, subsample=0.75, n_estimators=250, max_features=20,\n",
    "                                  max_depth=6, learning_rate=0.03)\n",
    "rf = RandomForestClassifier(n_estimators=100, min_samples_split=2, min_samples_leaf=1, \n",
    "                            max_features='sqrt', max_depth=6, bootstrap=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 線性迴歸預測檔 (結果有部分隨機, 請以 Kaggle 計算的得分為準, 以下模型同理)\n",
    "lr.fit(train_X, train_Y)\n",
    "lr_pred = lr.predict_proba(test_X)[:,1]\n",
    "sub = pd.DataFrame({'PassengerId': ids, 'Survived': lr_pred})\n",
    "sub['Survived'] = sub['Survived'].map(lambda x:1 if x>0.5 else 0) \n",
    "sub.to_csv('titanic_lr.csv', index=False) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 梯度提升機預測檔 \n",
    "gdbt.fit(train_X, train_Y)\n",
    "gdbt_pred = gdbt.predict_proba(test_X)[:,1]\n",
    "sub = pd.DataFrame({'PassengerId': ids, 'Survived': gdbt_pred})\n",
    "sub['Survived'] = sub['Survived'].map(lambda x:1 if x>0.5 else 0) \n",
    "sub.to_csv('titanic_gdbt.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 隨機森林預測檔\n",
    "rf.fit(train_X, train_Y)\n",
    "rf_pred = rf.predict_proba(test_X)[:,1]\n",
    "sub = pd.DataFrame({'PassengerId': ids, 'Survived': rf_pred})\n",
    "sub['Survived'] = sub['Survived'].map(lambda x:1 if x>0.5 else 0) \n",
    "sub.to_csv('titanic_rf.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 作業\n",
    "* 雖然同樣是混合泛化，分類預測其實與回歸預測有相當多的差異性，\n",
    "因為鐵達尼預測的結果是 '生存/死亡'，輸出不是 0 就是 1  \n",
    "因此要用權重混合時，需要以以機率的形式混合，因此我們在作業前幾格當中，先幫各位同學把預測值寫成了機率的形式  \n",
    "(請同學把下列程式完成，並將結果提交到 Kaggle 網站看看結果)\n",
    "\n",
    "* 但是光是這樣，分類問題的混合泛化就能比單模預測還要好嗎?  \n",
    "已經快要期中考了，這裡請同學挑戰看看，還有沒有什麼方法可以改進混合泛化的結果?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 混合泛化預測檔 \n",
    "\"\"\"\n",
    "Your Code Here\n",
    "\"\"\"\n",
    "blending_pred = lr_pred*0.06  + gdbt_pred*0.24 + rf_pred*0.7\n",
    "sub = pd.DataFrame({'PassengerId': ids, 'Survived': blending_pred})\n",
    "sub['Survived'] =  sub['Survived'].map(lambda x:1 if x>0.5 else 0)\n",
    "sub.to_csv('titanic_blending.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.2 64-bit",
   "language": "python",
   "name": "python38264bit3aa85e905e81479f963b5b188efc9345"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}