Day_030_HW-Feature_Selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 作業 : (Kaggle)鐵達尼生存預測\n",
    "https://www.kaggle.com/c/titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [作業目標]\n",
    "- 試著調整特徵篩選的門檻值, 觀察會有什麼影響效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [作業重點]\n",
    "- 調整相關係數過濾法的篩選門檻, 看看篩選結果的影響 (In[5]~In[8], Out[5]~Out[8])\n",
    "- 調整L1 嵌入法篩選門檻, 看看篩選結果的影響 (In[9]~In[11], Out[9]~Out[11])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 做完特徵工程前的所有準備 (與前範例相同)\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import copy\n",
    "from sklearn.preprocessing import LabelEncoder, MinMaxScaler\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.linear_model import LogisticRegression\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Ticket</th>\n",
       "      <th>Fare</th>\n",
       "      <th>Cabin</th>\n",
       "      <th>Embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Braund, Mr. Owen Harris</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>A/5 21171</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>PC 17599</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C85</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Heikkinen, Miss. Laina</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>STON/O2. 3101282</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>113803</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>C123</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Allen, Mr. William Henry</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>373450</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Pclass                                               Name  \\\n",
       "0         0       3                            Braund, Mr. Owen Harris   \n",
       "1         1       1  Cumings, Mrs. John Bradley (Florence Briggs Th...   \n",
       "2         1       3                             Heikkinen, Miss. Laina   \n",
       "3         1       1       Futrelle, Mrs. Jacques Heath (Lily May Peel)   \n",
       "4         0       3                           Allen, Mr. William Henry   \n",
       "\n",
       "      Sex   Age  SibSp  Parch            Ticket     Fare Cabin Embarked  \n",
       "0    male  22.0      1      0         A/5 21171   7.2500   NaN        S  \n",
       "1  female  38.0      1      0          PC 17599  71.2833   C85        C  \n",
       "2  female  26.0      0      0  STON/O2. 3101282   7.9250   NaN        S  \n",
       "3  female  35.0      1      0            113803  53.1000  C123        S  \n",
       "4    male  35.0      0      0            373450   8.0500   NaN        S  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = 'data/'\n",
    "df = pd.read_csv(data_path + 'titanic_train.csv')\n",
    "\n",
    "train_Y = df['Survived']\n",
    "df = df.drop(['PassengerId'] , axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHelJREFUeJzt3Xu0HWWZ5/HvLyFcYjAMARQkE0AiSCBEEsDAOEaUblyo0Ra5qmEaicwS22WP08KgGIMupZtuRSCMERkDLU24CEY6i0sHghElF+iEkGiAJtiEBDWAsALhknOe+aPeQ4pd+5yzd/aufSG/D6sWtaveqnpqn0o9+33fuigiMDMzyxvS7gDMzKzzODmYmVmBk4OZmRU4OZiZWYGTg5mZFTg5mJlZgZODmZkVODmYmVmBk4OZmRXs0O4AWuW1jY931a3gkw+b1u4Q6nb8TqPbHULdhofaHUJdDnitu+IFGL/T8+0OoW6Hrf1Fw190PeecYXsc0HF/WNccbLvVbYnBrJW2m5qDmVlL9fa0O4KGODmYmZWhZ0u7I2iIk4OZWQkietsdQkOcHMzMytDr5GBmZpVcczAzswJ3SJuZWYFrDmZmVil8tZKZmRV0eYe075A2MytD9NY+DELSCZLWSHpM0nlV5o+RtEDSQ5IWStq30fCdHMzMytDbU/swAElDgSuADwOHAKdJOqSi2CXANRExHpgJfKfR8J0czMzK0Lyaw1HAYxHxeES8ClwPTK0ocwiwII3fU2V+3ZwczMzK0LOl5kHSdEnLcsP03JreATyZ+7wuTctbAXwyjX8C2FXSqEbCr6lDWtIFwOlAD9ALfD4iFjeyYUkfAw6JiO82sp60rk0RMaLR9ZiZNU0dHdIRMRuY3c/sao8Prnwc+FeAyyWdCfwSeApo6HKpQZODpMnAR4AjIuIVSXsAO9ayckk7RETVACNiHjCvnmDNzLpFRNNuglsH5F+Wsi+w/o3bivXAXwFIGgF8MiIaepFGLc1KewMbI+KVFMTGiFgv6YmUKJA0SdLCND5D0mxJdwLXSFosaVzfylJP+kRJZ0q6XNLItK4haf5wSU9KGibpnZJul/SApEWSDk5l9pf0G0lLJV3UyBdgZlaK5vU5LAXGpvPejsCpVPywlrRH3zkUOB+4utHwa0kOdwKjJT0iaZak99ewzERgakScTtZ5cjKApL2BfSLigb6CKbutAPrW+1Hgjoh4jaya9cWImEhWbZqVylwKXBkRRwJP9xdEvh3vqmv+pYawzcyapLe39mEAqfXlXOAO4LfADRGxStLM1DwPMAVYI+kR4G3AtxsNf9BmpYjYJGki8D7gA8DcatfZVpgXEZvT+A3AXcA3yJLEjVXKzwVOIetlPxWYlapGxwA3Sq83ue2U/n8sWztfrgUu7if219vxuu01oWbW5Zr4+IyImA/Mr5h2YW78JuCmpm2QGjukI2s8WwgslLQSmEbW2dFX89i5YpEXc8s+JekZSePJEsDnq2xiHvAdSbuT1TruBt4C/DkiJvQXVi2xm5m1Rc9r7Y6gIYM2K0k6SNLY3KQJwO+BJ8hO5LD1V3x/rgf+DhgZESsrZ0bEJmAJWXPRbRHRExEvAGslfSrFIUmHp0XuI6thAJwx2D6YmbVck5qV2qWWPocRwBxJqyU9RHazxQzgm8ClkhaRXeI6kJvITuY3DFBmLvDp9P8+ZwBnSVoBrGLrjR1fAr4gaSkwsoZ9MDNrrSY+PqMdFLF9tM50W5/D5MOmtTuEuh2/0+jBC3WQ4VHt8vHOdsBr3Rfz+J0auqKyLQ5b+4uGv+iX7/tpzeecnY89o+P+sH4qq5lZGTq0uahWTg5mZiWILu+QdnIwMytDh/Yl1MrJwcysDG5WMjOzAtcczMyswDUHMzMrcM3BzMwKtjT0OoW2c3IwMyuDaw5mZlbgPgczMytwzcHMzApcc+gO3fYgu9+snNPuEOr2nnGntzuEup0/9MB2h1CXZ4e2O4L6veepB9sdQt2a0pXsmoNZd+q2xGBdxlcrmZlZQZe/DsHJwcysDO5zMDOzAicHMzMrcIe0mZkV9PS0O4KGODmYmZXBzUpmZlbg5GBmZgXuczAzs0rR6/sczMyskpuVzMyswFcrmZlZgWsOZmZW4ORQJKkHWJnW/1tgWkS81E/ZGcCmiLikjFjMzNqiyx+8N6Sk9W6OiAkRcSjwKnBOSdsxM+tMvb21Dx2orOSQtwg4EEDSZyU9JGmFpGsrC0o6W9LSNP9mScPT9E9JejhN/2WaNk7SEknL0zrHtmBfzMxq0xu1D4OQdIKkNZIek3ReP2VOlrRa0ipJ1zUafql9DpJ2AD4M3C5pHHABcGxEbJS0e5VFfhYRP0rLfgs4C7gMuBD4y4h4StJuqew5wKUR8VNJOwJd+I4sM3vTatLVSpKGAlcAxwPrgKWS5kXE6lyZscD5ZOfX5yTt1eh2y6o57CJpObAM+E/gx8BxwE0RsREgIp6tstyhkhZJWgmcAYxL0+8DfiLpbLYmgd8A/0fSV4ExEbG5cmWSpktaJmnZn156upn7Z2Y2oOjtrXkYxFHAYxHxeES8ClwPTK0oczZwRUQ8BxARf2w0/rL7HCZExBfTDgkYrP70E+DciDgM+CawM0BEnAN8DRgNLJc0KiKuAz4GbAbukHRc5coiYnZETIqISXsOf3vTds7MbFB1NCvlf8imYXpuTe8Ansx9Xpem5b0LeJek+yTdL+mERsNv5aWsC4BbJH0vIp6RtHuV2sOuwAZJw8hqDk8BSHpnRCwGFkv6KDBa0kjg8Yj4gaQDgPHA3a3bHTOzAdTxbKWImA3M7me2qi1S8XkHYCwwBdgXWCTp0Ij4c81BVFlhS0TEKknfBu5Nl7r+O3BmRbGvA4uB35NdCrtrmv4PqU1NZElmBXAe8GlJrwFPAzNL3wkzs1o179lK68haTfrsC6yvUub+iHgNWCtpDVmyWLqtGy0lOUTEiH6mzwHmVEybkRu/EriyynJ/VWV130mDmVnn2dK0x2csBcZK2p+sNeVU4PSKMrcCp5H1ze5B1sz0eCMb9R3SZmZlaNIjuyNii6RzgTvILsi5OrXEzASWRcS8NO8vJK0GeoD/HRHPNLJdJwczszI08ZHdETEfmF8x7cLceAB/m4amcHIwMytBDZeodjQnBzOzMvhlP2ZmVuDkYGZmBX7Zj5mZVfI7pM3MrMjJwczMCny1kpmZFbjmYGZmBU4OZmZWKXrcrNQVjt9p9OCFOsh7xlU+V6vz/fuqht9M2HJTDv9cu0Ooy5ghI9sdQt1+OmpKu0NoD9cczLpTtyUG6y6+lNXMzIqcHMzMrKC7uxycHMzMyhBbujs7ODmYmZWhu3ODk4OZWRncIW1mZkWuOZiZWSXXHMzMrMg1BzMzqxRb2h1BY5wczMxKEK45mJlZgZODmZlVcs3BzMwKuj05DGl3AH0kfUJSSDq43bGYmTUqelTz0Ik6JjkApwG/Ak5tdyBmZo2K3tqHTtQRyUHSCOBY4CxScpA0RNIsSask3SZpvqST0ryJku6V9ICkOyTt3cbwzcwKolc1D52oU/ocPg7cHhGPSHpW0hHAAcB+wGHAXsBvgaslDQMuA6ZGxJ8knQJ8G/jr9oRuZlbUqTWCWnVKcjgN+H4avz59HgbcGBG9wNOS7knzDwIOBe6SBDAU2FBtpZKmA9MBTtj9SCbsemBpO2BmlhfRmTWCWrU9OUgaBRwHHCopyE72AdzS3yLAqoiYPNi6I2I2MBvg/P1O7+4HnZhZV+n2mkMn9DmcBFwTEWMiYr+IGA2sBTYCn0x9D28DpqTya4A9JU0GkDRM0rh2BG5m1p/eHtU8dKK21xzImpC+WzHtZuDdwDrgYeARYDHwfES8mjqmfyBpJNk+fB9Y1bqQzcwG1qkdzbVqe3KIiClVpv0AsquYImJTanpaAqxM85cD/72VcZqZ1aOZyUHSCcClZM3uV0XEdyvmnwN8AegBNgHTI2J1I9tse3IYxG2SdgN2BC6KiKfbHZCZWS2iSb2ckoYCVwDHk7WmLJU0r+Lkf11E/N9U/mPAPwEnNLLdjk4O1WoVZmbdoIk1h6OAxyLicQBJ1wNTgdeTQ0S8kCv/FrKLehrS0cnBzKxbNfFS1ncAT+Y+rwOOriwk6QvA35K1tBzX6EY74WolM7M3nZ4e1TxImi5pWW6YnltVtSxTqBlExBUR8U7gq8DXGo3fNQczsxLUU3PI35NVxTpgdO7zvsD6AVZ3PXBlzRvvh2sOZmYlaOKzlZYCYyXtL2lHsufPzcsXkDQ29/FE4NFG43fNwcysBM26Wikitkg6F7iD7FLWqyNilaSZwLKImAecK+lDwGvAc8C0Rrfr5GBmVoJm3ucQEfOB+RXTLsyNf6lpG0ucHMzMStDT292t9k4OZmYlaFazUrs4OZiZlaDXj+w2M7NKfp+DmZkVuFmpSwzvsix+/tDue2vdlMM/1+4Q6rZwxVXtDqEuL0z7H+0OoW4PLtux3SG0hZuVzLpUtyUG6y6+WsnMzAq6vFXJycHMrAxuVjIzswJfrWRmZgW97Q6gQU4OZmYliKqvYegeTg5mZiXY4mYlMzOr5JqDmZkVuM/BzMwKXHMwM7MC1xzMzKygxzUHMzOr1MS3hLaFk4OZWQl6u7zm0JLHBkq6QNIqSQ9JWi7paElXSTokzd/Uz3LvlbQ4LfNbSTNaEa+ZWaOijqETlV5zkDQZ+AhwRES8ImkPYMeIqOXh/3OAkyNihaShwEFlxmpm1izd3iHdiprD3sDGiHgFICI2RsR6SQslTeorJOkfJT0oaYGkPdPkvYANabmeiFidys6QdK2kuyU9KunsFuyHmVnNeqWah07UiuRwJzBa0iOSZkl6f5UybwEejIgjgHuBb6Tp3wPWSLpF0ucl7ZxbZjxwIjAZuFDSPpUrlTRd0jJJy5ZteqypO2VmNpCeOoZOVHpyiIhNwERgOvAnYK6kMyuK9QJz0/g/A/8tLTsTmESWYE4Hbs8t8/OI2BwRG4F7gKOqbHt2REyKiEmTRnTfazfNrHv1qvahE7XkaqWI6AEWAgslrQSmDbZIbtn/AK6U9CPgT5JGVZbp57OZWdv4aqVBSDpI0tjcpAnA76vEcVIaPx34VVr2ROn1BrmxZDWwP6fPUyXtnJLFFGBpCeGbmW0TX600uBHAZZJ2A7YAj5E1Md2UK/MiME7SA8DzwClp+meA70l6KS17RkT0pHyxBPhX4L8CF0XE+hbsi5lZTTq1uahWpSeHiHgAOKbKrCm5MiPS6Ncrlj11gFU/EhHTGw7QzKwE3X4pq++QNjMrQY9rDq0XETPaHYOZ2UBcczAzswInBzMzK+jyV0i35sF7Zmbbm946hsFIOkHSGkmPSTqvyvydJM1N8xdL2q/R+J0czMxK0KzHZ6SHjl4BfBg4BDit74nWOWcBz0XEgWSPHbq40fidHMzMStDEx2ccBTwWEY9HxKvA9cDUijJTyZ5iDdk9ZB/M3UC8TZwczMxK0MRmpXcAT+Y+r0vTqpaJiC1kNxOPogFODmZmJagnOeSfIJ2G/A2+1WoAlU/dqKVMXXy1kplZCeo5M0fEbGB2P7PXAaNzn/cFKh8X1FdmnaQdgJHAs3WEUOCag5lZCZrY57AUGCtpf0k7AqcC8yrKzGPr065PAu6OCNcczMw6TbNe4hMRWySdC9wBDAWujohVkmYCyyJiHvBj4FpJj5HVGAZ6Ll1NtpvkcMBr3XVHyrND2x1B/cYMGdnuEOoybeL/4rJDG6p5t9xb5/y/dodQt7eO/0q7Q2iL3iY+jDsi5gPzK6ZdmBt/GfhU0zbIdpQczCp1W2Kw7uLHZ5iZWUGnvsSnVk4OZmYlcM3BzMwKtqi76w5ODmZmJeju1ODkYGZWCjcrmZlZQTMvZW0HJwczsxJ0d2pwcjAzK4WblczMrKCny+sOTg5mZiVwzcHMzArCNQczM6vkmoOZmRV0+6Wspb/sR1KPpOWSHpZ0o6ThTVjnmZIub0Z8ZmZliDqGTtSKN8FtjogJEXEo8CpwTq0LSurCtxqYmcEWouahE7X6NaGLgAMBJN0q6QFJq/Iv05a0SdJMSYuByZKOlPRrSSskLZG0ayq6j6TbJT0q6e9bvB9mZgOKOv7rRC1LDuml1x8GVqZJfx0RE4FJwN9IGpWmvwV4OCKOBpYAc4EvRcThwIeAzancBOAU4DDgFEn5F3D3bXO6pGWSlt390qNl7ZqZWUFvHUMnakVy2EXScmAZ8J9k7zqFLCGsAO4HRgNj0/Qe4OY0fhCwISKWAkTECxGxJc1bEBHPp9fjrQbGVG44ImZHxKSImHTc8LGVs83MStPtNYdWXK20OSIm5CdImkJWC5gcES9JWgjsnGa/HBF97+YW/ffXvJIb78FXXplZB+nUGkGtWt3n0Gck8FxKDAcD7+2n3O/I+haOBJC0a2qeMjPraD0RNQ+dqF0n2tuBcyQ9BKwha1oqiIhXJZ0CXCZpF7L+hg+1Lkwzs23T7fc5lJ4cImJElWmvkHVOD1o+9TdU1ix+koa+Mh9pNE4zs2bq1L6EWrmJxsysBN3e5+DkYGZWAjcrmZlZgZuVzMysoFOvQqqVk4OZWQncrGRmZgXukDYzswL3OZiZWYGblczMrCDcIW1mZpV6urzm0K4H75mZvan1EjUPjZC0u6S70ovP7pL0X6qUGZNerrY8vWBt0DdyOjmYmZUgImoeGnQe2fttxgIL0udKG4Bj0usTjgbOk7TPQCvdbpqVxu/0fLtDqMt7nnqw3SHU7aejprQ7hLr824PD2b13y+AFO8hbx3+l3SHU7YiHLml3CG3Rwg7pqcCUND4HWAh8NV8gIl7NfdyJGioGrjnYdqvbEoN1lxa+Ce5tEbEBIP1/r2qFJI1Or0l4Erg4ItYPtNLtpuZgZtZK9Tw+Q9J0YHpu0uyImJ2b/2/A26ssekGt24iIJ4HxqTnpVkk3RcQf+ivv5GBmVoJ6mpVSIpg9wPx+X3Im6Q+S9o6IDZL2Bv44yLbWS1oFvA+4qb9yblYyMytBq65WAuYB09L4NODnlQUk7Zvepkm6mulYsrdw9svJwcysBC28Wum7wPGSHgWOT5+RNEnSVanMu4HFklYA9wKXRMTKgVbqZiUzsxK06mqliHgG+GCV6cuAz6Xxu4Dx9azXycHMrAR+8J6ZmRX0RHc/tNvJwcysBH7wnpmZFfiR3WZmVuA+BzMzK+h1s5KZmVVyzcHMzAp8tdI2ktQD5O/Q+3hEPNGmcMzMmsrNSttuc3rxRF0kDY2InjICMjNrlm5vVuqoZytJ2k/SIkkPpuGYNH2KpHskXUeqbUj6tKQl6bV3P5Q0tK3Bm5nl9EbUPHSidtYcdpG0PI2vjYhPkD1q9viIeFnSWOBfgEmpzFHAoRGxVtK7gVOAYyPiNUmzgDOAa1q8D2ZmVXV7zaHTmpWGAZdLmgD0AO/KzVsSEWvT+AeBicBSSQC7UOUZ5vkXaFw46jBO2nVMc/fAzKwfPV3e+t1pVyt9GfgDcDhZk9fLuXkv5sYFzImI8wdaWf4FGiv3/2h3p3Ez6yrd/viMjupzAEYCGyKiF/gM0F8/wgLgJEl7AUjaXZKrBWbWMVr4sp9SdFpymAVMk3Q/WZPSi9UKRcRq4GvAnemF2XcBe7csSjOzQbTwZT+laFuzUkSMqDLtUd74Qorz0/SFwMKKsnOBueVFaGa27Tr1KqRadVqfg5nZm4KvVjIzswI/PsPMzAo6tS+hVk4OZmYlcJ+DmZkVuOZgZmYFnXr/Qq2cHMzMSuCag5mZFfhqJTMzK3CHtJmZFbhZyczMCnyHtJmZFbjmYNalnh2yA7v3bml3GPYm1e19Dur27NZukqanlwp1Dcdcvm6LFxyzvVGnvc+hG01vdwDbwDGXr9viBcdsOU4OZmZW4ORgZmYFTg6N68b2Tsdcvm6LFxyz5bhD2szMClxzMDOzgu0uOUi6QNIqSQ9JWi7p6Cas82OSzmtSfJvqKNuT9uFhSTdKGj5A2RmSvtKMGMsg6ROSQtLB7Y6lmmrHjaSrJB2S5lf9u0l6r6TFaZnfSprRonhrPjbqWOeZki5vRnw1bKsv/r5hv1Zs17barm6CkzQZ+AhwRES8ImkPYMcal90hIqreMRUR84B5zYu0ZpsjYgKApJ8C5wD/1IY4muE04FfAqcCM9obyRv0dNxHxuRoWnwOcHBErJA0FDioz1pxtPjYkDY2InjKDq8Hr8dejQ2J/U9jeag57Axsj4hWAiNgYEeslPZH+wSNpkqSFaXyGpNmS7gSuSb8Ax/WtTNJCSRP7flFJGpnWNSTNHy7pSUnDJL1T0u2SHpC0qO8XsqT9Jf1G0lJJFzWwb4uAA9M6P5t+4a6QdG1lQUlnp+2tkHRz369KSZ9KvzRXSPplmjZO0pL06+0hSWMbiLEqSSOAY4GzyJIDkoZImpV+rd8mab6kk9K8iZLuTd/lHZL2bnZMFfo7bhZKmpTbj3+U9KCkBZL2TJP3Ajak5XoiYnUqO0PStZLulvSopLNLjD9/bNyavrdVkl6/R0DSJkkzJS0GJks6UtKv07GwRNKuqeg+6Th+VNLflxhzgaT90r+dB9NwTJo+RdI9kq4DVqZpn84dtz9MidnqERHbzQCMAJYDjwCzgPen6U8Ae6TxScDCND4DeADYJX3+MvDNNL438EgaPxO4PI3/HPhAGj8FuCqNLwDGpvGjgbvT+Dzgs2n8C8CmOvZnU/r/Dmm7/xMYB6zJ7c/uuX35ShoflVvHt4AvpvGVwDvS+G7p/5cBZ6TxHfu+iyb/XT4N/DiN/xo4AjgJmE/2A+btwHNp2rBUZs/cd3x1m46bhcCkNB657+nC3PFwYYr9FuDzwM65v8cKYBdgD+BJYJ8mxlw4NiqOh12Ah/uOhRT/ybm/8+PAkenzW9N6zkzTRwI7A78HRpf0nfek73w5cEuaNjz3/Y0FlqXxKcCLwP7p87uBXwDD0udZpH9jHmoftqtmpYjYJGki8D7gA8BcDd5XMC8iNqfxG4C7gG8AJwM3Vik/l+yEdQ/Zr+BZ6ZfxMcCNkvrK7ZT+fyzwyTR+LXBxHbu0i6TlaXwR8GOyE9BNEbERICKerbLcoZK+BexGduK7I02/D/iJpBuAn6VpvwEukLQv8LOIeLSO+Gp1GvD9NH59+jwMuDEieoGnJd2T5h8EHArclb7LoaRf5mWp8bjpJfvbA/wz6fuLiJmpWecvgNPJ9m1KKvfzdGxtTvt3FHBrk8KudmwA/I2kT6Tx0WQn2WfITsY3p+kHARsiYmnahxcA0ve9ICKeT59XA2PIEluzVWtWGgZcLmlCivdduXlLImJtGv8gMBFYmmLeBfhjCTG+qW1XyQGyqj3ZL76FklYC04AtbG1i27likRdzyz4l6RlJ48kSwOerbGIe8B1Ju5MdoHcDbwH+XOVgf33V27g7hX9Ayv41DLa+nwAfj6wd/EzSySoizlHWQX8isFzShIi4LjU1nAjcIelzEXH3NsZbIGkUcBxZwgqyk32Q/dKuugiwKiImNyuGWvRz3Ay4SG7Z/wCulPQj4E9pn99Qpp/Pjah2bEwBPgRMjoiXlDWf9h3vL8fWtvqBjqFXcuM9tPYc8mXgD8DhZP9eX87NezE3LmBORJzfwtjedLarPgdJB1W0mU8gqxo/QXYih62/4vtzPfB3wMiIWFk5MyI2AUuAS4HbImtnfgFYK+lTKQ5JOjwtch+pnR04o/69KlgAnNx3AkpJqtKuwAZJw/LblPTOiFgcERcCG4HRkg4AHo+IH5AlvvFNiDHvJOCaiBgTEftFxGhgbdr+J1Pfw9vY+mt7DbCnsk5ilPXnjKu24mYZ4LjJG5L2BbIawq/Ssidqa3VxLNkJ9c/p81RJO6e/1RRgaQnh540EnkuJ4WDgvf2U+x1Z38KRAJJ2ldQJPyRHktVoeoHPkP2QqGYBcJKkvSD7NyBpTItifNPYrpIDWRPKHEmrJT0EHELW9vtN4FJJi8j+8Q7kJrKT+Q0DlJlL1o4+NzftDOAsSSuAVcDUNP1LwBckLSU7+BsSEauAbwP3pm1Vu0Ll68Bisiay3+Wm/4OklZIeBn5J1iZ+CvBwaqI4GLim0RgrnEaxlnAzsA+wjqxd/Icp3ucj4lWyk/DFaf+WkzXZlam/4ybvRWCcpAfIakIz0/TPAGvS93ctWb9E3zG2BPhX4H7goohYX+5ucDuwQ9qHi9J2C9J3fApwWfqO76JYo26HWcA0SfeTNSm9WK1QZJ3+XwPuTPt6F1kfodXBd0hbx5I0IrX3jyI7kR4bEU+3O65mUHa/w6aIuKTdsZhV0wlVRbP+3CZpN7KrZy56syQGs27gmoOZmRVsb30OZmZWAycHMzMrcHIwM7MCJwczMytwcjAzswInBzMzK/j/1rPTbPWNfK4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 計算df整體相關係數, 並繪製成熱圖\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "corr = df.corr()\n",
    "sns.heatmap(corr)\n",
    "plt.show()\n",
    "\n",
    "# 出現 <Figure size 640x480 with 2 Axes> 就全部重新 run 一次\n",
    "# -0.3 表示較不相關"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 記得刪除 Survived\n",
    "df = df.drop(['Survived'] , axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5 Numeric Features : ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#只取 int64, float64 兩種數值型欄位, 存於 num_features 中\n",
    "num_features = []\n",
    "for dtype, feature in zip(df.dtypes, df.columns):\n",
    "    if dtype == 'float64' or dtype == 'int64':\n",
    "        num_features.append(feature)\n",
    "print(f'{len(num_features)} Numeric Features : {num_features}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>SibSp</th>\n",
       "      <th>Parch</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pclass   Age  SibSp  Parch     Fare\n",
       "0       3  22.0      1      0   7.2500\n",
       "1       1  38.0      1      0  71.2833\n",
       "2       3  26.0      0      0   7.9250\n",
       "3       1  35.0      1      0  53.1000\n",
       "4       3  35.0      0      0   8.0500"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 削減文字型欄位, 只剩數值型欄位\n",
    "df = df[num_features]\n",
    "df = df.fillna(-1)\n",
    "MMEncoder = MinMaxScaler()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 作業1\n",
    "* 鐵達尼生存率預測中，試著變更兩種以上的相關係數門檻值，觀察預測能力是否提升?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7038635542329971"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 原始特徵 + 邏輯斯迴歸\n",
    "train_X = MMEncoder.fit_transform(df)\n",
    "estimator = LogisticRegression()\n",
    "cross_val_score(estimator, train_X, train_Y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pclass', 'Age', 'Parch', 'Fare']\n"
     ]
    }
   ],
   "source": [
    "# 篩選相關係數大於 0.05 或小於 -0.05 的特徵\n",
    "\n",
    "high_list = list(corr[(corr['Survived']>0.05) | (corr['Survived']<-0.05)].index)\n",
    "high_list.pop(0)\n",
    "print(high_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7049872206659582"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 0.05特徵 + 邏輯斯迴歸\n",
    "train_X = MMEncoder.fit_transform(df[high_list])\n",
    "cross_val_score(estimator, train_X, train_Y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pclass', 'Fare']\n"
     ]
    }
   ],
   "source": [
    "# 篩選相關係數大於 0.1 或小於 -0.1 的特徵\n",
    "high_list = list(corr[(corr['Survived']>0.1) | (corr['Survived']<-0.1)].index)\n",
    "high_list.pop(0)\n",
    "print(high_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.6791567235397566"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 0.1特徵 + 邏輯斯迴歸\n",
    "train_X = MMEncoder.fit_transform(df[high_list])\n",
    "cross_val_score(estimator, train_X, train_Y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 由上述不同門檻可以發現, 相關係數有時候確實能成功提升準確度, 但篩選過頭會有反效果\n",
    "# 門檻的決定沒有一定的準則, 因此這個方法並不那麼好用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作業2\n",
    "* 續上題，使用 L1 Embedding 做特徵選擇(自訂門檻)，觀察預測能力是否提升?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.39126312, -0.1355752 , -0.        ,  0.03068895,  0.        ])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "L1_Reg = Lasso(alpha=0.005)\n",
    "train_X = MMEncoder.fit_transform(df)\n",
    "L1_Reg.fit(train_X, train_Y)\n",
    "L1_Reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Pclass', 'Age', 'Parch']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import compress\n",
    "L1_mask = list((L1_Reg.coef_>0) | (L1_Reg.coef_<0))\n",
    "L1_list = list(compress(list(df), list(L1_mask)))\n",
    "L1_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "D:\\Anaconda\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.6993628951406613"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# L1_Embedding 特徵 + 線性迴歸\n",
    "train_X = MMEncoder.fit_transform(df[L1_list])\n",
    "cross_val_score(estimator, train_X, train_Y, cv=5).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 在作業與範例中, 效果並不如顯著\n",
    "# 而且與相關係數法類似, L1 Embedding 也需要調整 alpha 值, 且沒有一定法則\n",
    "# 所以也並非好用的特徵篩選方式\n",
    "# 2019/10/13"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}