Day3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiple Linear Regression\n",
    "\n",
    "Multiple linear regression attempts to model the relationship between two or more features and a response by fitting a linear equation to observed data. The steps to perform multiple linear regression are almost similar to that of simple linear regression. The difference lies in the evaluation. You can use it to find out which factor has the highest impact on the predicted output and how different variables relate to each other.\n",
    "\n",
    "多元線性迴歸透過用一個線性方程式來配適觀察資料，這線性方程式是在兩個以上（包括兩個）的特徵和響應之間構建的一個關係。多元線性回歸的實現步驟和簡單線性迴歸很相似，在評價部分有所不同。可以用它來找出在預測結果上哪個因素影響力最大，以及不同變量是如何相戶觀連的。\n",
    "\n",
    "\n",
    "y = $b_0$ + $b_1$$x_1$ + $b_2$$x_2$ ..... + $b_n$$x_n$\n",
    "\n",
    "$x_n$：multiple independent variables  \n",
    "y：Dependent variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assumptions\n",
    "\n",
    "For a successful regression analysis, It's essential to validate these assumptions.\n",
    "\n",
    "1. Linearity：The relationship between dependent and independent variables should be Linear.\n",
    "2. Homoscedasticity (constant variance) of the errors should be maintained.\n",
    "3. Multivariate Normality：Multiple regression assumes that the residuals are normally distributed.\n",
    "4. Lack of Multicollinearity：It is assumed that there is little or no multicollinearity in the data. Multicollinearity occurs when the features (or independent vairiables) are not independent of each other.\n",
    "\n",
    "要有一個成功的迴歸分析，驗證這些假釋是必要的。\n",
    "\n",
    "1. 線性：自變量與應變量的關係應該是線性的(即特徵值和預測值是具有線性相關)。\n",
    "2. 應保持誤差的等分散性(相同的變異數)。\n",
    "3. 多元常態分佈：多元回歸假定殘差符合常態分佈。\n",
    "4. 缺少多重共線性：假設資料中很少或沒有多重共線性。 多重共線性是發生在，當特徵(或獨立的可變元素)彼此不相互獨立時。\n",
    "\n",
    "\n",
    "在多元迴歸分析中，直線模式具有下列三種推論假設：(1)相同的變異數(constant variance)；(2)獨立性(independence)；(3)常態分配(normal distribution)。\n",
    "\n",
    "Homoscedasticity(等分散性) 可參考連結的解釋 http://terms.naer.edu.tw/detail/1311574/\n",
    "\n",
    "Multicollinearity(多元共線性) 可參考連結的解釋 http://terms.naer.edu.tw/detail/1304718/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note\n",
    "\n",
    "Having too many variables could potentially cause our model to become lessaccurate, especially if certain variables have no effect on the outcome or have a significant effect on other vaiables. There are vaious methods to select the appropriate vaiable like - \n",
    "1. Forward Selection\n",
    "2. Backward Elimination\n",
    "3. Bi-directional Compaision\n",
    "\n",
    "過多的變量可能會降低模型的精準度，特別是如果存有一些對結果無關的變量，或是存在對其他變量造成很大影響的變量。有一些選擇合適變量的方法 -\n",
    "1. 向前選擇法\n",
    "2. 向後選擇法 (也稱 向後剔除法 / 向後消除法)\n",
    "3. 向前向後法 是結合上面向前及向後選擇法，先用向前選擇法篩選一遍，再用向後選擇法篩選一遍，直到最後無論怎麼篩選，模型變量都不再發生變化，就算結束了"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dummy Variables\n",
    "\n",
    "Using categorical data in Multiple Regression Models is a powerful method to include non-numeric data types into a regression model. Categorical data refers to data values which repressent categories - data values with a fixed and unordered number of values, for instance, gender (male/female). In a regression model, these values containing values such as 1 or 0 repressenting the presence or absence of the categorical value.\n",
    "\n",
    "| Gender |\n",
    "| :---: |\n",
    "| Female |\n",
    "| Female |\n",
    "| Male |\n",
    "| Female |\n",
    "| Male |\n",
    "| Male |\n",
    "| Male |  \n",
    "\n",
    "=>\n",
    "\n",
    "| Male | Female |\n",
    "| --- | --- |\n",
    "| 0 | 1 |\n",
    "| 0 | 1 |\n",
    "| 1 | 0 |\n",
    "| 0 | 1 |\n",
    "| 1 | 0 |\n",
    "| 1 | 0 |\n",
    "| 1 | 0 |\n",
    "\n",
    "在多元迴歸模型中，當遇到資料集內是有非數值數據類型時，使用分類數據是一個非常有效的方法。分類數據是指代表類別的數據值，是離散數據 - 其數值個數（分類屬性）具有固定和無序數值的數據值，例如性別（男/女 兩類）。在一個迴歸模型中，這些分類值可以用虛擬變量來表示，該變量通常取如 1 或 0 這樣的值來表示肯定或否定類型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dummy Variable Trap\n",
    "\n",
    "The Dummy Variable trap is a scenario in which two or more variables are highly correlated; in simple terms, one variable can be predicted from the others. Intuitively, there is a duplicate category: if we dropped the male category it is inherently defined in the female category (zero female value indicate male, and vice-versa). The solution to the dummy vaiable trap is to drop one of the categorical vaiables - if there are m number of categories, use m-1 in the model, the value left out cna be thought of as the reference value.\n",
    "\n",
    "$D_2$ = 1 - $D_1$  \n",
    "\n",
    "$D_2$：Dummy variable  \n",
    "$D_1$：Dummy variable\n",
    "\n",
    "y = $b_0$ + $b_1$$x_1$ + $b_2$$x_2$ + $b_3$$D_1$\n",
    "\n",
    "虛擬變量陷阱是指兩個以上（包括兩個）變量之間高度相關的情況。簡單來說，就是存在一個能夠被其他變量預測出來的變量，舉一個存在重複類別（變量）的直觀例子：假如捨棄男性類別，則該類別也可以透過女性類別來定義(女性值為 0 時，可表示成男性，反之亦然)。解決虛擬變量陷阱的方法是，捨棄其中一個分類變量，即類別變量減去 1：假如有 m 個類別，那在模型建構時，取 m-1 個虛擬變量，減去的那個變量可看作是參照值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Data Preprocessing\n",
    "\n",
    "* Import the Libraries.\n",
    "* Import the DataSet.\n",
    "* Check for Missing Data.\n",
    "* Encode Categorical Data.\n",
    "* Make Dummy Variables if necessary and avoid dummy variable trap.\n",
    "* Feature Scaling will be taken care by the Library we will use for Simple Linear Regression Model.\n",
    "\n",
    "\n",
    "* 載入相關的套件\n",
    "* 載入資料集\n",
    "* 檢查遺失資料\n",
    "* 將分類資料進行編碼\n",
    "* 如有必要時，編輯虛擬變數並注意避免虛擬變數陷阱\n",
    "* 特徵縮放將用簡單線性迴歸模型的相關套件來執行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import The Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing The Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>R&amp;D Spend</th>\n",
       "      <th>Administration</th>\n",
       "      <th>Marketing Spend</th>\n",
       "      <th>State</th>\n",
       "      <th>Profit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>165349.20</td>\n",
       "      <td>136897.80</td>\n",
       "      <td>471784.10</td>\n",
       "      <td>New York</td>\n",
       "      <td>192261.83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>162597.70</td>\n",
       "      <td>151377.59</td>\n",
       "      <td>443898.53</td>\n",
       "      <td>California</td>\n",
       "      <td>191792.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>153441.51</td>\n",
       "      <td>101145.55</td>\n",
       "      <td>407934.54</td>\n",
       "      <td>Florida</td>\n",
       "      <td>191050.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>144372.41</td>\n",
       "      <td>118671.85</td>\n",
       "      <td>383199.62</td>\n",
       "      <td>New York</td>\n",
       "      <td>182901.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>142107.34</td>\n",
       "      <td>91391.77</td>\n",
       "      <td>366168.42</td>\n",
       "      <td>Florida</td>\n",
       "      <td>166187.94</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   R&D Spend  Administration  Marketing Spend       State     Profit\n",
       "0  165349.20       136897.80        471784.10    New York  192261.83\n",
       "1  162597.70       151377.59        443898.53  California  191792.06\n",
       "2  153441.51       101145.55        407934.54     Florida  191050.39\n",
       "3  144372.41       118671.85        383199.62    New York  182901.99\n",
       "4  142107.34        91391.77        366168.42     Florida  166187.94"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = pd.read_csv('./datasets/50_Startups.csv')\n",
    "dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[165349.2, 136897.8, 471784.1, 'New York'],\n",
       "       [162597.7, 151377.59, 443898.53, 'California'],\n",
       "       [153441.51, 101145.55, 407934.54, 'Florida'],\n",
       "       [144372.41, 118671.85, 383199.62, 'New York'],\n",
       "       [142107.34, 91391.77, 366168.42, 'Florida']], dtype=object)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = dataset.iloc[:, :-1].values\n",
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([192261.83, 191792.06, 191050.39, 182901.99, 166187.94])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y = dataset.iloc[:, 4].values\n",
    "Y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding Categorical Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder, OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "labelencoder = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[165349.2, 136897.8, 471784.1, 2],\n",
       "       [162597.7, 151377.59, 443898.53, 0],\n",
       "       [153441.51, 101145.55, 407934.54, 1],\n",
       "       [144372.41, 118671.85, 383199.62, 2],\n",
       "       [142107.34, 91391.77, 366168.42, 1]], dtype=object)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:, 3] = labelencoder.fit_transform(X[:, 3])\n",
    "X[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "onehotencoder  = OneHotEncoder(categorical_features = [3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jasonch/anaconda3/envs/tf_cpu/lib/python3.6/site-packages/sklearn/preprocessing/_encoders.py:363: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values.\n",
      "If you want the future behaviour and silence this warning, you can specify \"categories='auto'\".\n",
      "In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly.\n",
      "  warnings.warn(msg, FutureWarning)\n",
      "/Users/jasonch/anaconda3/envs/tf_cpu/lib/python3.6/site-packages/sklearn/preprocessing/_encoders.py:385: DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20 and will be removed in 0.22. You can use the ColumnTransformer instead.\n",
      "  \"use the ColumnTransformer instead.\", DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.6534920e+05,\n",
       "        1.3689780e+05, 4.7178410e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 0.0000000e+00, 1.6259770e+05,\n",
       "        1.5137759e+05, 4.4389853e+05],\n",
       "       [0.0000000e+00, 1.0000000e+00, 0.0000000e+00, 1.5344151e+05,\n",
       "        1.0114555e+05, 4.0793454e+05],\n",
       "       [0.0000000e+00, 0.0000000e+00, 1.0000000e+00, 1.4437241e+05,\n",
       "        1.1867185e+05, 3.8319962e+05],\n",
       "       [0.0000000e+00, 1.0000000e+00, 0.0000000e+00, 1.4210734e+05,\n",
       "        9.1391770e+04, 3.6616842e+05]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = onehotencoder.fit_transform(X).toarray()\n",
    "X[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Avoiding Dummy Variable Trap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.0000000e+00, 1.0000000e+00, 1.6534920e+05, 1.3689780e+05,\n",
       "        4.7178410e+05],\n",
       "       [0.0000000e+00, 0.0000000e+00, 1.6259770e+05, 1.5137759e+05,\n",
       "        4.4389853e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 1.5344151e+05, 1.0114555e+05,\n",
       "        4.0793454e+05],\n",
       "       [0.0000000e+00, 1.0000000e+00, 1.4437241e+05, 1.1867185e+05,\n",
       "        3.8319962e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 1.4210734e+05, 9.1391770e+04,\n",
       "        3.6616842e+05]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X[:, 1:]\n",
    "X[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting The Dataset Into The Training set and Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.0000000e+00, 0.0000000e+00, 5.5493950e+04, 1.0305749e+05,\n",
       "        2.1463481e+05],\n",
       "       [0.0000000e+00, 1.0000000e+00, 4.6014020e+04, 8.5047440e+04,\n",
       "        2.0551764e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 7.5328870e+04, 1.4413598e+05,\n",
       "        1.3405007e+05],\n",
       "       [0.0000000e+00, 0.0000000e+00, 4.6426070e+04, 1.5769392e+05,\n",
       "        2.1079767e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 9.1749160e+04, 1.1417579e+05,\n",
       "        2.9491957e+05]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 5)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.0000000e+00, 0.0000000e+00, 6.6051520e+04, 1.8264556e+05,\n",
       "        1.1814820e+05],\n",
       "       [0.0000000e+00, 0.0000000e+00, 1.0067196e+05, 9.1790610e+04,\n",
       "        2.4974455e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 1.0191308e+05, 1.1059411e+05,\n",
       "        2.2916095e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 2.7892920e+04, 8.4710770e+04,\n",
       "        1.6447071e+05],\n",
       "       [1.0000000e+00, 0.0000000e+00, 1.5344151e+05, 1.0114555e+05,\n",
       "        4.0793454e+05]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 5)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Fitting Multiple Linear Regression To The Training Set\n",
    "\n",
    "This step is exactly the same as for simple linear regression. To fit the dataset into the model we will use LinearRegression class form sklearn.linear_model library. Then we make an object regressor of LinearRegression Class. Now we will fit the regressor object into out dataset using fit() method of LinearRegression Class.\n",
    "\n",
    "這步驟和簡單線性迴歸模型的處理一樣。用 sklearn.linear_model 的 LinearRegression 在資料集上訓練模型。建立一個 LinearRegression Class 的 regressor 物件。使用 LinearRegression Class 的 fit()，在資料集上配適進行訓練。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressor = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,\n",
       "         normalize=False)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regressor.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Prediction The Test Set Results\n",
    "\n",
    "Now we will predict the observations from our test set. We will save the output in a vector Y_pred. To predict the result we use predict() method of LinearRegression Class on the regressor we trained in the previous step.\n",
    "\n",
    "在測試集上進行預測，並檢視結果。將輸出結果儲存在向量 Y_pred。使用前一步驟訓練時所用的 LinearRegression 的 regressor，用其 predict() 來預測結果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = regressor.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([103015.20159796, 132582.27760815, 132447.73845175,  71976.09851258,\n",
       "       178537.48221056])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred[:5]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}