W02_Prakash Panicker_qq328457 (3).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your name:\n",
    "\n",
    "<pre> Nitin Prakash Panicker</pre>\n",
    "\n",
    "### Collaborators:\n",
    "\n",
    "<pre> </pre>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the housing data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-122.23</td>\n",
       "      <td>37.88</td>\n",
       "      <td>41.0</td>\n",
       "      <td>880.0</td>\n",
       "      <td>129.0</td>\n",
       "      <td>322.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>8.3252</td>\n",
       "      <td>452600.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-122.22</td>\n",
       "      <td>37.86</td>\n",
       "      <td>21.0</td>\n",
       "      <td>7099.0</td>\n",
       "      <td>1106.0</td>\n",
       "      <td>2401.0</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>8.3014</td>\n",
       "      <td>358500.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-122.24</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1467.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>496.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>7.2574</td>\n",
       "      <td>352100.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1274.0</td>\n",
       "      <td>235.0</td>\n",
       "      <td>558.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>5.6431</td>\n",
       "      <td>341300.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-122.25</td>\n",
       "      <td>37.85</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1627.0</td>\n",
       "      <td>280.0</td>\n",
       "      <td>565.0</td>\n",
       "      <td>259.0</td>\n",
       "      <td>3.8462</td>\n",
       "      <td>342200.0</td>\n",
       "      <td>NEAR BAY</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0    -122.23     37.88                41.0        880.0           129.0   \n",
       "1    -122.22     37.86                21.0       7099.0          1106.0   \n",
       "2    -122.24     37.85                52.0       1467.0           190.0   \n",
       "3    -122.25     37.85                52.0       1274.0           235.0   \n",
       "4    -122.25     37.85                52.0       1627.0           280.0   \n",
       "\n",
       "   population  households  median_income  median_house_value ocean_proximity  \n",
       "0       322.0       126.0         8.3252            452600.0        NEAR BAY  \n",
       "1      2401.0      1138.0         8.3014            358500.0        NEAR BAY  \n",
       "2       496.0       177.0         7.2574            352100.0        NEAR BAY  \n",
       "3       558.0       219.0         5.6431            341300.0        NEAR BAY  \n",
       "4       565.0       259.0         3.8462            342200.0        NEAR BAY  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import tarfile\n",
    "from six.moves import urllib\n",
    "\n",
    "DOWNLOAD_ROOT = \"https://raw.githubusercontent.com/ageron/handson-ml/master/\"\n",
    "HOUSING_PATH = os.path.join(\"datasets\", \"housing\")\n",
    "HOUSING_URL = DOWNLOAD_ROOT + \"datasets/housing/housing.tgz\"\n",
    "\n",
    "def fetch_housing_data(housing_url=HOUSING_URL, housing_path=HOUSING_PATH):\n",
    "    if not os.path.isdir(housing_path):\n",
    "        os.makedirs(housing_path)\n",
    "    tgz_path = os.path.join(housing_path, \"housing.tgz\")\n",
    "    urllib.request.urlretrieve(housing_url, tgz_path)\n",
    "    housing_tgz = tarfile.open(tgz_path)\n",
    "    housing_tgz.extractall(path=housing_path)\n",
    "    housing_tgz.close()\n",
    "\n",
    "fetch_housing_data()\n",
    "import pandas as pd\n",
    "\n",
    "def load_housing_data(housing_path=HOUSING_PATH):\n",
    "    csv_path = os.path.join(housing_path, \"housing.csv\")\n",
    "    return pd.read_csv(csv_path)\n",
    "\n",
    "housing = load_housing_data()\n",
    "housing.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a full pipeline for data analysis following the example of the module notebook.\n",
    " Hint: the main change is the algorithm used (Lasso regression)\n",
    "\n",
    "If you want to learn more about the Lasso regression, see resources below:\n",
    "- http://scikit-learn.org/stable/modules/linear_model.html#lasso\n",
    "- https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Considerations for building a pipeline:\n",
    "\n",
    "- Split data into training and testing sets (1 mark)\n",
    "- Convert all categorical data to one-hot vectors below inside a pipeline (2 marks)\n",
    "- Normalize all non-categorical data inside the same pipeline (2 marks)\n",
    "- Perform Lasso-based regression (1 mark)\n",
    "- Perform a grid search using a variety of values for $\\alpha$ between 0 and 1 where *housing_labels* is the output and all other features are the input (2 marks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 20640 entries, 0 to 20639\n",
      "Data columns (total 10 columns):\n",
      " #   Column              Non-Null Count  Dtype  \n",
      "---  ------              --------------  -----  \n",
      " 0   longitude           20640 non-null  float64\n",
      " 1   latitude            20640 non-null  float64\n",
      " 2   housing_median_age  20640 non-null  float64\n",
      " 3   total_rooms         20640 non-null  float64\n",
      " 4   total_bedrooms      20433 non-null  float64\n",
      " 5   population          20640 non-null  float64\n",
      " 6   households          20640 non-null  float64\n",
      " 7   median_income       20640 non-null  float64\n",
      " 8   median_house_value  20640 non-null  float64\n",
      " 9   ocean_proximity     20640 non-null  object \n",
      "dtypes: float64(9), object(1)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "housing.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXHklEQVR4nO3df5Dc9X3f8ecrwsYyZyFR7Kui01QkUUkA1di6oWoZMncBByVoEH+UjjzEiIaOOgxxcEeZcmpm2ukfajXTktYMgY4GXMRAfdUQM2hM5FhVuPFkBowlgn0WWEUxiiykSIkDmMMM7tFX/9gP8kbau9uTVrt3/rweMzv7/b6/3+9+33s/Xvvdz353V7aJiIg6/FyvG4iIiO5J6EdEVCShHxFRkYR+RERFEvoRERW5oNcNzOTSSy/1ihUrTs2/8847XHTRRb1rqE3ps7PSZ2elz86ai33u37//b2x//IwFtuf0ZfXq1W727LPPej5In52VPjsrfXbWXOwT2OcWmZrhnYiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIisz5j2GYj1aMPMPmVZPcMfJM1/d9eNtNXd9nRMwfOdKPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqMmPoS7pc0ktNlx9J+oKkSyTtkfRquV7StM0WSYckHZR0Y1N9taTxsux+STpfdywiIs40Y+jbPmj7attXA6uBHwNPASPAXtsrgb1lHklXABuAK4G1wIOSFpSbewjYBKwsl7WdvTsRETGd2Q7vXA/8he2/BNYDO0p9B3BLmV4PjNp+z/ZrwCHgGklLgUW2nytf2vtY0zYREdEFauRvmytLXwJetP2ApDdtL25a9obtJZIeAJ63/XipPwLsBg4D22zfUOrXAffaXtdiP5toPCOgv79/9ejo6KllExMT9PX1zf6edtH462/RvxBOvNv9fa9advGs1p8PP09In52WPjtrLvY5PDy83/bg6fW2P3BN0oeBm4EtM63aouZp6mcW7e3AdoDBwUEPDQ2dWjY2Nkbz/Fx0R/nAtfvGu/95dodvG5rV+vPh5wnps9PSZ2fNlz5hdsM7v0HjKP9EmT9Rhmwo1ydL/SiwvGm7AeBYqQ+0qEdERJfMJvQ/C3y5aX4XsLFMbwSebqpvkHShpMtovGD7gu3jwNuS1pSzdm5v2iYiIrqgrfEHSR8FPgP8q6byNmCnpDuBI8CtALYPSNoJvAxMAnfbfr9scxfwKLCQxjj/7g7ch4iIaFNboW/7x8DfO632Qxpn87RafyuwtUV9H3DV7NuMiIhOyDtyIyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiLtfjH6YuBhGt9va+C3gYPA/wJWAIeBf277jbL+FuBO4H3gd23/Samv5qdfjP7HwD223bF7E6wYeWZW629eNckds9ymlcPbbjrn24iI86/dI/0vAl+z/cvAJ4FXgBFgr+2VwN4yj6QrgA3AlcBa4EFJC8rtPARsAlaWy9oO3Y+IiGjDjKEvaRHwq8AjALZ/YvtNYD2wo6y2A7ilTK8HRm2/Z/s14BBwjaSlwCLbz5Wj+8eatomIiC7QTKMrkq4GtgMv0zjK3w/cA7xue3HTem/YXiLpAeB524+X+iPAbhpDQNts31Dq1wH32l7XYp+baDwjoL+/f/Xo6OipZRMTE/T19Z31He6G8dffon8hnHi3153MrFN9rlp28bnfyDTmw+8d0menpc+zNzw8vN/24On1dsb0LwA+DXze9jclfZEylDMFtah5mvqZRXs7jQcaBgcHPTQ0dGrZ2NgYzfNz0R0jz7B51ST3jbf1kklPdarPw7cNnXsz05gPv3dIn52WPjuvnTH9o8BR298s80/SeBA4UYZsKNcnm9Zf3rT9AHCs1Ada1CMioktmDH3bfwX8QNLlpXQ9jaGeXcDGUtsIPF2mdwEbJF0o6TIaL9i+YPs48LakNZIE3N60TUREdEG7z+s/Dzwh6cPA94F/QeMBY6ekO4EjwK0Atg9I2knjgWESuNv2++V27uKnp2zuLpeIiOiStkLf9kvAGS8I0Djqb7X+VmBri/o+Guf6R0RED+QduRERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUJKEfEVGRhH5EREUS+hERFUnoR0RUpK3Ql3RY0riklyTtK7VLJO2R9Gq5XtK0/hZJhyQdlHRjU311uZ1Dku4vX5AeERFdMpsj/WHbV9v+4LtyR4C9tlcCe8s8kq4ANgBXAmuBByUtKNs8BGwCVpbL2nO/CxER0a5zGd5ZD+wo0zuAW5rqo7bfs/0acAi4RtJSYJHt52wbeKxpm4iI6IJ2Q9/A1yXtl7Sp1PptHwco158o9WXAD5q2PVpqy8r06fWIiOiSC9pc71rbxyR9Atgj6XvTrNtqnN7T1M+8gcYDyyaA/v5+xsbGTi2bmJj4O/Nz0eZVk/QvbFzPdZ3q83z/TubD7x3SZ6elz85rK/RtHyvXJyU9BVwDnJC01PbxMnRzsqx+FFjetPkAcKzUB1rUW+1vO7AdYHBw0ENDQ6eWjY2N0Tw/F90x8gybV01y33i7j6m906k+D982dO7NTGM+/N4hfXZa+uy8GYd3JF0k6WMfTAO/DnwX2AVsLKttBJ4u07uADZIulHQZjRdsXyhDQG9LWlPO2rm9aZuIiOiCdg7x+oGnytmVFwD/0/bXJH0L2CnpTuAIcCuA7QOSdgIvA5PA3bbfL7d1F/AosBDYXS4REdElM4a+7e8Dn2xR/yFw/RTbbAW2tqjvA66afZsREdEJeUduRERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVKTt0Je0QNKfS/pqmb9E0h5Jr5brJU3rbpF0SNJBSTc21VdLGi/L7lf5tvWIiOiO2Rzp3wO80jQ/Auy1vRLYW+aRdAWwAbgSWAs8KGlB2eYhYBOwslzWnlP3ERExK22FvqQB4Cbg4abyemBHmd4B3NJUH7X9nu3XgEPANZKWAotsP2fbwGNN20RERBeokb8zrCQ9Cfwn4GPA79leJ+lN24ub1nnD9hJJDwDP23681B8BdgOHgW22byj164B7ba9rsb9NNJ4R0N/fv3p0dPTUsomJCfr6+s72/nbF+Otv0b8QTrzb605m1qk+Vy27+NxvZBrz4fcO6bPT0ufZGx4e3m978PT6BTNtKGkdcNL2fklDbeyr1Ti9p6mfWbS3A9sBBgcHPTT0092OjY3RPD8X3THyDJtXTXLf+Iw/3p7rVJ+Hbxs692amMR9+75A+Oy19dl47/+3XAjdL+k3gI8AiSY8DJyQttX28DN2cLOsfBZY3bT8AHCv1gRb1iIjokhnH9G1vsT1gewWNF2j/1PZvAbuAjWW1jcDTZXoXsEHShZIuo/GC7Qu2jwNvS1pTztq5vWmbiIjognN5Xr8N2CnpTuAIcCuA7QOSdgIvA5PA3bbfL9vcBTwKLKQxzr/7HPYfERGzNKvQtz0GjJXpHwLXT7HeVmBri/o+4KrZNhkREZ2Rd+RGRFQkoR8RUZGEfkRERRL6EREVSehHRFQkoR8RUZGEfkRERRL6EREVSehHRFQkoR8RUZGEfkRERRL6EREVSehHRFQkoR8RUZGEfkRERRL6EREVSehHRFQkoR8RUZGEfkRERWYMfUkfkfSCpG9LOiDpP5T6JZL2SHq1XC9p2maLpEOSDkq6sam+WtJ4WXa/JJ2fuxUREa20c6T/HvBrtj8JXA2slbQGGAH22l4J7C3zSLoC2ABcCawFHpS0oNzWQ8AmYGW5rO3gfYmIiBnMGPpumCizHyoXA+uBHaW+A7ilTK8HRm2/Z/s14BBwjaSlwCLbz9k28FjTNhER0QVq5O8MKzWO1PcDvwT8oe17Jb1pe3HTOm/YXiLpAeB524+X+iPAbuAwsM32DaV+HXCv7XUt9reJxjMC+vv7V4+Ojp5aNjExQV9f39ne364Yf/0t+hfCiXd73cnMOtXnqmUXn/uNTGM+/N4hfXZa+jx7w8PD+20Pnl6/oJ2Nbb8PXC1pMfCUpKumWb3VOL2nqbfa33ZgO8Dg4KCHhoZOLRsbG6N5fi66Y+QZNq+a5L7xtn68PdWxPsffOffbmMbmVe9z35+13sfhbTed133Pxnz4+4T02WnzpU+Y5dk7tt8ExmiMxZ8oQzaU65NltaPA8qbNBoBjpT7Qoh4REV3Sztk7Hy9H+EhaCNwAfA/YBWwsq20Eni7Tu4ANki6UdBmNF2xfsH0ceFvSmnLWzu1N20RERBe087x+KbCjjOv/HLDT9lclPQfslHQncAS4FcD2AUk7gZeBSeDuMjwEcBfwKLCQxjj/7k7emYiImN6MoW/7O8CnWtR/CFw/xTZbga0t6vuA6V4PiIiI8yjvyI2IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKJPQjIiqS0I+IqEhCPyKiIgn9iIiKzP0vcT0HK0ae6XULERFzSo70IyIqktCPiKhIQj8ioiIzhr6k5ZKelfSKpAOS7in1SyTtkfRquV7StM0WSYckHZR0Y1N9taTxsux+STo/dysiIlpp50h/Eths+1eANcDdkq4ARoC9tlcCe8s8ZdkG4EpgLfCgpAXlth4CNgEry2VtB+9LRETMYMbQt33c9otl+m3gFWAZsB7YUVbbAdxSptcDo7bfs/0acAi4RtJSYJHt52wbeKxpm4iI6AI18rfNlaUVwDeAq4Ajthc3LXvD9hJJDwDP23681B8BdgOHgW22byj164B7ba9rsZ9NNJ4R0N/fv3p0dPTUsomJCfr6+trqd/z1t9q+b53WvxBOvNuz3bftZ6HPVcsu7m4z05jN32cvpc/Omot9Dg8P77c9eHq97fP0JfUBfwR8wfaPphmOb7XA09TPLNrbge0Ag4ODHhoaOrVsbGyM5vnp3NHD8/Q3r5rkvvG5/zaIn4U+D9821N1mpjGbv89eSp+dNV/6hDbP3pH0IRqB/4Ttr5TyiTJkQ7k+WepHgeVNmw8Ax0p9oEU9IiK6pJ2zdwQ8Arxi+w+aFu0CNpbpjcDTTfUNki6UdBmNF2xfsH0ceFvSmnKbtzdtExERXdDO8/prgc8B45JeKrV/C2wDdkq6EzgC3Apg+4CkncDLNM78udv2+2W7u4BHgYU0xvl3d+h+REREG2YMfdt/RuvxeIDrp9hmK7C1RX0fjReBIyKiB/KO3IiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIqktCPiKjIjKEv6UuSTkr6blPtEkl7JL1arpc0Ldsi6ZCkg5JubKqvljRelt0vaarv3Y2IiPNkxi9GBx4FHgAea6qNAHttb5M0UubvlXQFsAG4Evh54H9L+oe23wceAjYBzwN/DKwFdnfqjkS9Vow805P9Ht52U0/2G3EuZjzSt/0N4G9PK68HdpTpHcAtTfVR2+/Zfg04BFwjaSmwyPZztk3jAeQWIiKiq9TI4BlWklYAX7V9VZl/0/bipuVv2F4i6QHgeduPl/ojNI7mDwPbbN9Q6tcB99peN8X+NtF4VkB/f//q0dHRU8smJibo6+tr686Nv/5WW+udD/0L4cS7Pdt929Ln2Vu17OIzarP5++yl9NlZc7HP4eHh/bYHT6+3M7wzG63G6T1NvSXb24HtAIODgx4aGjq1bGxsjOb56dzRo6f9AJtXTXLfeKd/vJ2XPs/e4duGzqjN5u+zl9JnZ82XPuHsz945UYZsKNcnS/0osLxpvQHgWKkPtKhHREQXnW3o7wI2lumNwNNN9Q2SLpR0GbASeMH2ceBtSWvKWTu3N20TERFdMuPzZUlfBoaASyUdBf49sA3YKelO4AhwK4DtA5J2Ai8Dk8Dd5cwdgLtonAm0kMY4f87ciYjoshlD3/Znp1h0/RTrbwW2tqjvA66aVXcREdFReUduRERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERVJ6EdEVCShHxFRkYR+RERFEvoRERWZW59VGzGPtPrGrs2rJrvykd751q44WznSj4ioSEI/IqIiCf2IiIok9CMiKpLQj4ioSEI/IqIiOWUzYh5qdbrobJztqaU5VXT+y5F+RERFun6kL2kt8EVgAfCw7W3d7iEizs65PsOYrQ+ekeQZRud0NfQlLQD+EPgMcBT4lqRdtl/uZh8RMb90+8Fmts7HO7HP1wNdt4d3rgEO2f6+7Z8Ao8D6LvcQEVEt2e7ezqR/Bqy1/S/L/OeAf2z7d05bbxOwqcxeDhxsWnwp8DddaPdcpc/OSp+dlT47ay72+Q9sf/z0YrfH9NWidsajju3twPaWNyDtsz3Y6cY6LX12VvrsrPTZWfOlT+j+8M5RYHnT/ABwrMs9RERUq9uh/y1gpaTLJH0Y2ADs6nIPERHV6urwju1JSb8D/AmNUza/ZPvALG+m5bDPHJQ+Oyt9dlb67Kz50md3X8iNiIjeyjtyIyIqktCPiKjIvAp9SWslHZR0SNJIr/tpRdJySc9KekXSAUn39Lqn6UhaIOnPJX21171MRdJiSU9K+l75uf6TXvfUiqR/XX7n35X0ZUkf6XVPAJK+JOmkpO821S6RtEfSq+V6SS97LD216vM/l9/7dyQ9JWlxL3ssPZ3RZ9Oy35NkSZf2ord2zJvQb/oIh98ArgA+K+mK3nbV0iSw2favAGuAu+donx+4B3il103M4IvA12z/MvBJ5mC/kpYBvwsM2r6KxokKG3rb1SmPAmtPq40Ae22vBPaW+V57lDP73ANcZfsfAf8H2NLtplp4lDP7RNJyGh8xc6TbDc3GvAl95slHONg+bvvFMv02jYBa1tuuWpM0ANwEPNzrXqYiaRHwq8AjALZ/YvvN3nY1pQuAhZIuAD7KHHkPiu1vAH97Wnk9sKNM7wBu6WpTLbTq0/bXbU+W2edpvLenp6b4eQL8V+Df0OINp3PJfAr9ZcAPmuaPMkfD9AOSVgCfAr7Z206m9N9o/JH+v143Mo1fAP4a+B9lGOphSRf1uqnT2X4d+C80jvKOA2/Z/npvu5pWv+3j0DhQAT7R437a8dvA7l430Yqkm4HXbX+7173MZD6Fflsf4TBXSOoD/gj4gu0f9bqf00laB5y0vb/XvczgAuDTwEO2PwW8w9wYivg7ypj4euAy4OeBiyT9Vm+7+tkh6fdpDJ0+0eteTifpo8DvA/+u1720Yz6F/rz5CAdJH6IR+E/Y/kqv+5nCtcDNkg7TGCr7NUmP97allo4CR21/8GzpSRoPAnPNDcBrtv/a9v8FvgL80x73NJ0TkpYClOuTPe5nSpI2AuuA2zw331j0izQe7L9d/p8GgBcl/f2edjWF+RT68+IjHCSJxvjzK7b/oNf9TMX2FtsDtlfQ+Fn+qe05d2Rq+6+AH0i6vJSuB+bi9y8cAdZI+mj5G7ieOfiCc5NdwMYyvRF4uoe9TKl86dK9wM22f9zrflqxPW77E7ZXlP+no8Cny9/unDNvQr+8mPPBRzi8Auw8i49w6IZrgc/ROHJ+qVx+s9dNzXOfB56Q9B3gauA/9rifM5RnIk8CLwLjNP635sRb8yV9GXgOuFzSUUl3AtuAz0h6lcYZJz3/Brsp+nwA+Biwp/wv/feeNsmUfc4b+RiGiIiKzJsj/YiIOHcJ/YiIiiT0IyIqktCPiKhIQj8ioiIJ/YiIiiT0IyIq8v8BGTt6an/Xox0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# As per experts (business), median income is a very important attribute to predict housing prices. \n",
    "# Since median income is a continuous numerical attribute, we need to first create an income category attribute. \n",
    "\n",
    "#  Let’s look at the median income histogram more closely\n",
    "housing[\"median_income\"].hist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following code creates an income category \n",
    "# attribute by dividing the median income by 1.5 (to limit the number of income cate‐gories), \n",
    "# and rounding up using ceil (to have discrete categories), and then merging\n",
    "# all the categories greater than 5 into category 5\n",
    "\n",
    "housing[\"income_cat\"] = np.ceil(housing[\"median_income\"] / 1.5)\n",
    "housing[\"income_cat\"].where(housing[\"income_cat\"] < 5, 5.0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0    7236\n",
       "2.0    6581\n",
       "4.0    3639\n",
       "5.0    2362\n",
       "1.0     822\n",
       "Name: income_cat, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing[\"income_cat\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train-Test split based on stratified sampling technique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we are ready to do stratified sampling based on the income category. \n",
    "# For this we can use Scikit-Learn’s StratifiedShuffleSplit class\n",
    "\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "split = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=42)\n",
    "for train_index, test_index in split.split(housing, housing[\"income_cat\"]):\n",
    "    strat_train_set = housing.loc[train_index]\n",
    "    strat_test_set = housing.loc[test_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.0    0.350581\n",
       "2.0    0.318847\n",
       "4.0    0.176308\n",
       "5.0    0.114438\n",
       "1.0    0.039826\n",
       "Name: income_cat, dtype: float64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# income category proportions in the full housing dataset\n",
    "\n",
    "housing[\"income_cat\"].value_counts() / len(housing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now you should remove the income_cat attribute so the data is back to its original state:\n",
    "\n",
    "for set_ in (strat_train_set, strat_test_set):\n",
    "    set_.drop(\"income_cat\", axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing = strat_train_set.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attribute combinations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the total number of rooms in a district is not very useful if you don’t know how many\n",
    "# households there are. What you really want is the number of rooms per household.\n",
    "# Similarly, the total number of bedrooms by itself is not very useful: you probably\n",
    "# want to compare it to the number of rooms. And the population per household also\n",
    "# seems like an interesting attribute combination to look at. Let’s create these new\n",
    "# attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "housing[\"rooms_per_household\"] = housing[\"total_rooms\"]/housing[\"households\"]\n",
    "housing[\"bedrooms_per_room\"] = housing[\"total_bedrooms\"]/housing[\"total_rooms\"]\n",
    "housing[\"population_per_household\"]=housing[\"population\"]/housing[\"households\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "median_house_value          1.000000\n",
       "median_income               0.687160\n",
       "rooms_per_household         0.146285\n",
       "total_rooms                 0.135097\n",
       "housing_median_age          0.114110\n",
       "households                  0.064506\n",
       "total_bedrooms              0.047689\n",
       "population_per_household   -0.021985\n",
       "population                 -0.026920\n",
       "longitude                  -0.047432\n",
       "latitude                   -0.142724\n",
       "bedrooms_per_room          -0.259984\n",
       "Name: median_house_value, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And now let’s look at the correlation matrix again:\n",
    "corr_matrix = housing.corr()\n",
    "corr_matrix[\"median_house_value\"].sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare data for machine learning algorithms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let’s revert to a clean training set (by copying strat_train_set once again),\n",
    "# and let’s separate the predictors and the labels since we don’t necessarily want to apply\n",
    "# the same transformations to the predictors and the target values\n",
    "\n",
    "housing = strat_train_set.drop(\"median_house_value\", axis=1)\n",
    "housing_labels = strat_train_set[\"median_house_value\"].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill missing values for total_bedrooms using sci-kit learn Imputer\n",
    "try:\n",
    "    from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+\n",
    "except ImportError:\n",
    "    from sklearn.preprocessing import Imputer as SimpleImputer\n",
    "\n",
    "imputer = SimpleImputer(strategy=\"median\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since the median can only be computed on numerical attributes, we need to create a\n",
    "# copy of the data without the text attribute ocean_proximity:\n",
    "\n",
    "housing_num = housing.drop(\"ocean_proximity\", axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleImputer(add_indicator=False, copy=True, fill_value=None,\n",
       "              missing_values=nan, strategy='median', verbose=0)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now you can fit the imputer instance to the training data using the fit() method:\n",
    "imputer.fit(housing_num)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transfer the training set:\n",
    "# Now we can use this “trained” imputer to transform the training set by replacing missing values by the learned medians:\n",
    "\n",
    "X = imputer.transform(housing_num)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The result is a plain Numpy array containing the transformed features. If you want to\n",
    "# put it back into a Pandas DataFrame, it’s simple:\n",
    "\n",
    "housing_tr = pd.DataFrame(X, columns=housing_num.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-121.89</td>\n",
       "      <td>37.29</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1568.0</td>\n",
       "      <td>351.0</td>\n",
       "      <td>710.0</td>\n",
       "      <td>339.0</td>\n",
       "      <td>2.7042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-121.93</td>\n",
       "      <td>37.05</td>\n",
       "      <td>14.0</td>\n",
       "      <td>679.0</td>\n",
       "      <td>108.0</td>\n",
       "      <td>306.0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>6.4214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-117.20</td>\n",
       "      <td>32.77</td>\n",
       "      <td>31.0</td>\n",
       "      <td>1952.0</td>\n",
       "      <td>471.0</td>\n",
       "      <td>936.0</td>\n",
       "      <td>462.0</td>\n",
       "      <td>2.8621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-119.61</td>\n",
       "      <td>36.31</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1847.0</td>\n",
       "      <td>371.0</td>\n",
       "      <td>1460.0</td>\n",
       "      <td>353.0</td>\n",
       "      <td>1.8839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-118.59</td>\n",
       "      <td>34.23</td>\n",
       "      <td>17.0</td>\n",
       "      <td>6592.0</td>\n",
       "      <td>1525.0</td>\n",
       "      <td>4459.0</td>\n",
       "      <td>1463.0</td>\n",
       "      <td>3.0347</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0    -121.89     37.29                38.0       1568.0           351.0   \n",
       "1    -121.93     37.05                14.0        679.0           108.0   \n",
       "2    -117.20     32.77                31.0       1952.0           471.0   \n",
       "3    -119.61     36.31                25.0       1847.0           371.0   \n",
       "4    -118.59     34.23                17.0       6592.0          1525.0   \n",
       "\n",
       "   population  households  median_income  \n",
       "0       710.0       339.0         2.7042  \n",
       "1       306.0       113.0         6.4214  \n",
       "2       936.0       462.0         2.8621  \n",
       "3      1460.0       353.0         1.8839  \n",
       "4      4459.0      1463.0         3.0347  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_tr.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling Text and Categorical Attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17606</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18632</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14650</th>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3230</th>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3555</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19480</th>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8879</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13685</th>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4937</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4861</th>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ocean_proximity\n",
       "17606       <1H OCEAN\n",
       "18632       <1H OCEAN\n",
       "14650      NEAR OCEAN\n",
       "3230           INLAND\n",
       "3555        <1H OCEAN\n",
       "19480          INLAND\n",
       "8879        <1H OCEAN\n",
       "13685          INLAND\n",
       "4937        <1H OCEAN\n",
       "4861        <1H OCEAN"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_cat = housing[['ocean_proximity']]\n",
    "housing_cat.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [4.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ordinal_encoder = OrdinalEncoder()\n",
    "housing_cat_encoded = ordinal_encoder.fit_transform(housing_cat)\n",
    "housing_cat_encoded[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array(['<1H OCEAN', 'INLAND', 'ISLAND', 'NEAR BAY', 'NEAR OCEAN'],\n",
       "       dtype=object)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ordinal_encoder.categories_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., 0., 0.],\n",
       "       [1., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0., 1.],\n",
       "       ...,\n",
       "       [0., 1., 0., 0., 0.],\n",
       "       [1., 0., 0., 0., 0.],\n",
       "       [0., 0., 0., 1., 0.]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_encoder = OneHotEncoder(sparse=False)\n",
    "housing_cat_1hot = cat_encoder.fit_transform(housing_cat)\n",
    "housing_cat_1hot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Transformers to  add extra attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "# get the right column indices: safer than hard-coding indices 3, 4, 5, 6\n",
    "rooms_ix, bedrooms_ix, population_ix, household_ix = [\n",
    "    list(housing.columns).index(col)\n",
    "    for col in (\"total_rooms\", \"total_bedrooms\", \"population\", \"households\")]\n",
    "\n",
    "def add_extra_features(X, add_bedrooms_per_room=True):\n",
    "    rooms_per_household = X[:, rooms_ix] / X[:, household_ix]\n",
    "    population_per_household = X[:, population_ix] / X[:, household_ix]\n",
    "    if add_bedrooms_per_room:\n",
    "        bedrooms_per_room = X[:, bedrooms_ix] / X[:, rooms_ix]\n",
    "        return np.c_[X, rooms_per_household, population_per_household,\n",
    "                     bedrooms_per_room]\n",
    "    else:\n",
    "        return np.c_[X, rooms_per_household, population_per_household]\n",
    "\n",
    "attr_adder = FunctionTransformer(add_extra_features, validate=False,\n",
    "                                 kw_args={\"add_bedrooms_per_room\": False})\n",
    "housing_extra_attribs = attr_adder.fit_transform(housing.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-121.89</td>\n",
       "      <td>37.29</td>\n",
       "      <td>38</td>\n",
       "      <td>1568</td>\n",
       "      <td>351</td>\n",
       "      <td>710</td>\n",
       "      <td>339</td>\n",
       "      <td>2.7042</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>4.62537</td>\n",
       "      <td>2.0944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-121.93</td>\n",
       "      <td>37.05</td>\n",
       "      <td>14</td>\n",
       "      <td>679</td>\n",
       "      <td>108</td>\n",
       "      <td>306</td>\n",
       "      <td>113</td>\n",
       "      <td>6.4214</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.00885</td>\n",
       "      <td>2.70796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-117.2</td>\n",
       "      <td>32.77</td>\n",
       "      <td>31</td>\n",
       "      <td>1952</td>\n",
       "      <td>471</td>\n",
       "      <td>936</td>\n",
       "      <td>462</td>\n",
       "      <td>2.8621</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.22511</td>\n",
       "      <td>2.02597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-119.61</td>\n",
       "      <td>36.31</td>\n",
       "      <td>25</td>\n",
       "      <td>1847</td>\n",
       "      <td>371</td>\n",
       "      <td>1460</td>\n",
       "      <td>353</td>\n",
       "      <td>1.8839</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.23229</td>\n",
       "      <td>4.13598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-118.59</td>\n",
       "      <td>34.23</td>\n",
       "      <td>17</td>\n",
       "      <td>6592</td>\n",
       "      <td>1525</td>\n",
       "      <td>4459</td>\n",
       "      <td>1463</td>\n",
       "      <td>3.0347</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>4.50581</td>\n",
       "      <td>3.04785</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  longitude latitude housing_median_age total_rooms total_bedrooms population  \\\n",
       "0   -121.89    37.29                 38        1568            351        710   \n",
       "1   -121.93    37.05                 14         679            108        306   \n",
       "2    -117.2    32.77                 31        1952            471        936   \n",
       "3   -119.61    36.31                 25        1847            371       1460   \n",
       "4   -118.59    34.23                 17        6592           1525       4459   \n",
       "\n",
       "  households median_income ocean_proximity rooms_per_household  \\\n",
       "0        339        2.7042       <1H OCEAN             4.62537   \n",
       "1        113        6.4214       <1H OCEAN             6.00885   \n",
       "2        462        2.8621      NEAR OCEAN             4.22511   \n",
       "3        353        1.8839          INLAND             5.23229   \n",
       "4       1463        3.0347       <1H OCEAN             4.50581   \n",
       "\n",
       "  population_per_household  \n",
       "0                   2.0944  \n",
       "1                  2.70796  \n",
       "2                  2.02597  \n",
       "3                  4.13598  \n",
       "4                  3.04785  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_extra_attribs = pd.DataFrame(\n",
    "    housing_extra_attribs,\n",
    "    columns=list(housing.columns)+[\"rooms_per_household\", \"population_per_household\"])\n",
    "housing_extra_attribs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation Pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "num_pipeline = Pipeline([\n",
    "        ('imputer', SimpleImputer(strategy=\"median\")),\n",
    "        ('attribs_adder', FunctionTransformer(add_extra_features, validate=False)),\n",
    "        ('std_scaler', StandardScaler()),\n",
    "    ])\n",
    "\n",
    "housing_num_tr = num_pipeline.fit_transform(housing_num)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.15604281,  0.77194962,  0.74333089, ..., -0.31205452,\n",
       "        -0.08649871,  0.15531753],\n",
       "       [-1.17602483,  0.6596948 , -1.1653172 , ...,  0.21768338,\n",
       "        -0.03353391, -0.83628902],\n",
       "       [ 1.18684903, -1.34218285,  0.18664186, ..., -0.46531516,\n",
       "        -0.09240499,  0.4222004 ],\n",
       "       ...,\n",
       "       [ 1.58648943, -0.72478134, -1.56295222, ...,  0.3469342 ,\n",
       "        -0.03055414, -0.52177644],\n",
       "       [ 0.78221312, -0.85106801,  0.18664186, ...,  0.02499488,\n",
       "         0.06150916, -0.30340741],\n",
       "       [-1.43579109,  0.99645926,  1.85670895, ..., -0.22852947,\n",
       "        -0.09586294,  0.10180567]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_num_tr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_attribs = list(housing_num)\n",
    "cat_attribs = [\"ocean_proximity\"]\n",
    "\n",
    "full_pipeline = ColumnTransformer([\n",
    "        (\"num\", num_pipeline, num_attribs),\n",
    "        (\"cat\", OneHotEncoder(), cat_attribs),\n",
    "    ])\n",
    "\n",
    "#housing_prepared = full_pipeline.fit_transform(housing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.15604281,  0.77194962,  0.74333089, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-1.17602483,  0.6596948 , -1.1653172 , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 1.18684903, -1.34218285,  0.18664186, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       ...,\n",
       "       [ 1.58648943, -0.72478134, -1.56295222, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.78221312, -0.85106801,  0.18664186, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-1.43579109,  0.99645926,  1.85670895, ...,  0.        ,\n",
       "         1.        ,  0.        ]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_prepared = full_pipeline.fit_transform(housing)\n",
    "housing_prepared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16512, 16)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_prepared.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/srv/conda/envs/notebook/lib/python3.7/site-packages/sklearn/linear_model/_coordinate_descent.py:476: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 6387975536398.07, tolerance: 22103585513.76512\n",
      "  positive)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000,\n",
       "      normalize=False, positive=False, precompute=False, random_state=None,\n",
       "      selection='cyclic', tol=0.0001, warm_start=False)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "# Write your code here:\n",
    "lasso = Lasso(alpha=0.1)\n",
    "lasso.fit(housing_prepared, housing_labels)\n",
    "#lasso_error = calculate_rmse(lasso, 'Lasso Reg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-55649.69623971 -56715.68160335  13732.97013106  -1930.40246579\n",
      "   7328.92163613 -45707.69311818  45453.38957137  74714.08082073\n",
      "   6604.70432957   1042.78205511   9250.09312996  14558.19020894\n",
      " -22644.85155224 142106.58083121  10093.85890506  17931.50098746]\n"
     ]
    }
   ],
   "source": [
    "print(lasso.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [210644.9970783  317768.2531878  210955.57732393  59219.78605864\n",
      " 189746.677462  ]\n"
     ]
    }
   ],
   "source": [
    "# let's try the full preprocessing pipeline on a few training instances\n",
    "some_data = housing.iloc[:5]\n",
    "some_labels = housing_labels.iloc[:5]\n",
    "some_data_prepared = full_pipeline.transform(some_data)\n",
    "\n",
    "print(\"Predictions:\", lasso.predict(some_data_prepared))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels: [286600.0, 340600.0, 196900.0, 46300.0, 254500.0]\n"
     ]
    }
   ],
   "source": [
    "print(\"Labels:\", list(some_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.15604281,  0.77194962,  0.74333089, -0.49323393, -0.44543821,\n",
       "        -0.63621141, -0.42069842, -0.61493744, -0.31205452, -0.08649871,\n",
       "         0.15531753,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ],\n",
       "       [-1.17602483,  0.6596948 , -1.1653172 , -0.90896655, -1.0369278 ,\n",
       "        -0.99833135, -1.02222705,  1.33645936,  0.21768338, -0.03353391,\n",
       "        -0.83628902,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ],\n",
       "       [ 1.18684903, -1.34218285,  0.18664186, -0.31365989, -0.15334458,\n",
       "        -0.43363936, -0.0933178 , -0.5320456 , -0.46531516, -0.09240499,\n",
       "         0.4222004 ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ],\n",
       "       [-0.01706767,  0.31357576, -0.29052016, -0.36276217, -0.39675594,\n",
       "         0.03604096, -0.38343559, -1.04556555, -0.07966124,  0.08973561,\n",
       "        -0.19645314,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ],\n",
       "       [ 0.49247384, -0.65929936, -0.92673619,  1.85619316,  2.41221109,\n",
       "         2.72415407,  2.57097492, -0.44143679, -0.35783383, -0.00419445,\n",
       "         0.2699277 ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_data_prepared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "68628.19880703553"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "housing_predictions = lasso.predict(housing_prepared)\n",
    "lasso_mse = mean_squared_error(housing_labels, housing_predictions)\n",
    "lasso_rmse = np.sqrt(lasso_mse)\n",
    "lasso_rmse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49439.9232897164"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "lasso_mae = mean_absolute_error(housing_labels, housing_predictions)\n",
    "lasso_mae"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LassoCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg = LassoCV(cv=5, random_state=0).fit(housing_prepared, housing_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6474231155660904"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg.score(housing_prepared,housing_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([207449.99701277, 314578.37873388, 209065.64635044, ...,\n",
       "       100176.40361568, 215597.92256334, 275672.95313929])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg.predict(housing_prepared)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grid Search for Algorithm Tuning\n",
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "# load the diabetes datasets\n",
    "dataset = datasets.load_diabetes()\n",
    "# prepare a range of alpha values to test\n",
    "alphas = np.array([1,0.1,0.01,0.001,0.0001,0])\n",
    "# create and fit a ridge regression model, testing each alpha\n",
    "model = Lasso()\n",
    "grid = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))\n",
    "grid.fit(housing_prepared, housing_labels)\n",
    "print(grid)\n",
    "# summarize the results of the grid search\n",
    "print(grid.best_score_)\n",
    "print(grid.best_estimator_.alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Why is it necessary to normalize all continuous variables before performing Lasso? (OPTIONAL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre> WRITE RESPONSE HERE </pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions (2 marks)\n",
    "What values of $\\alpha$ make Lasso perform best? Does it perform as well on the housing data as the linear regressor from the lectures? Why do you think this is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lasso regression tends to completely eliminate the weights of the least important features and automatically performs feature selection and outputs a sparse model. Smaller values of $\\alpha$ would make lasso perform better. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read appendix B\n",
    "\n",
    "- Reflect on your last data project, read appendix B. Then, write down a few of the checklist items that your last data project could have used. If you have not yet done a data project, then write down a few of the items that you found most interesting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}