setup_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Description\n",
    "\n",
    "http://archive.ics.uci.edu/ml/datasets/Adult\n",
    "\n",
    "\n",
    "**Donor**: \n",
    "\n",
    "    Ronny Kohavi and Barry Becker \n",
    "    Data Mining and Visualization \n",
    "    Silicon Graphics. \n",
    "    e-mail: ronnyk '@' live.com for questions. \n",
    "\n",
    "\n",
    "**Data Set Information**:\n",
    "\n",
    "    Extraction was done by Barry Becker from the 1994 Census database. A set of reasonably clean records was extracted using the following conditions: ((AAGE>16) && (AGI>100) && (AFNLWGT>1)&& (HRSWK>0)) \n",
    "\n",
    "    Prediction task is to determine whether a person makes over 50K a year. \n",
    "\n",
    "\n",
    "----\n",
    "\n",
    "**Listing of attributes**: \n",
    "\n",
    "**class**: >50K, <=50K. \n",
    "\n",
    "**workclass**: Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked. \n",
    "\n",
    "\n",
    "**education**: Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool. \n",
    "\n",
    "**marital-status**: Married-civ-spouse, Divorced, Never-married, Separated, Widowed, Married-spouse-absent, Married-AF-spouse. \n",
    "\n",
    "**occupation**: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces. \n",
    "\n",
    "**relationship**: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried. \n",
    "\n",
    "**race**: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black. \n",
    "\n",
    "**sex**: Female, Male. \n",
    "\n",
    "**native-country**: United-States, Cambodia, England, Puerto-Rico, Canada, Germany, Outlying-US(Guam-USVI-etc), India, Japan, Greece, South, China, Cuba, Iran, Honduras, Philippines, Italy, Poland, Jamaica, Vietnam, Mexico, Portugal, Ireland, France, Dominican-Republic, Laos, Ecuador, Taiwan, Haiti, Columbia, Hungary, Guatemala, Nicaragua, Scotland, Thailand, Yugoslavia, El-Salvador, Trinadad&Tobago, Peru, Hong, Holand-Netherlands.\n",
    "\n",
    "**age**: continuous. \n",
    "\n",
    "**capital-gain**: continuous. \n",
    "\n",
    "**capital-loss**: continuous. \n",
    "\n",
    "**hours-per-week**: continuous. \n",
    "\n",
    "**education-num**: continuous. \n",
    "\n",
    "**fnlwgt**: continuous. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import  Libraries \n",
    "These libraries are going to be used in all notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.391867Z",
     "start_time": "2019-11-21T02:03:42.386998Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input dataset (further split into training and validation dasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.453031Z",
     "start_time": "2019-11-21T02:03:42.395208Z"
    }
   },
   "outputs": [],
   "source": [
    "path = 'adult.data'\n",
    "\n",
    "df = pd.read_csv(path, index_col=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test dataset\n",
    "\n",
    "This should remain untouched during training or validation/calibration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.486796Z",
     "start_time": "2019-11-21T02:03:42.455821Z"
    }
   },
   "outputs": [],
   "source": [
    "path = 'adult.test'\n",
    "\n",
    "df_t = pd.read_csv(path, index_col=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discard non-informative dimensions in both datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.503481Z",
     "start_time": "2019-11-21T02:03:42.489251Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.drop('fnlwgt', 1)\n",
    "df = df.drop('education-num', 1)\n",
    "\n",
    "df_t = df_t.drop(u'fnlwgt', 1)\n",
    "df_t = df_t.drop(u'education-num', 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discard information that may lead to gender or race bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.519586Z",
     "start_time": "2019-11-21T02:03:42.505548Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.drop('race', 1)\n",
    "df = df.drop('sex', 1)\n",
    "\n",
    "df_t = df_t.drop(u'race', 1)\n",
    "df_t = df_t.drop(u'sex', 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hidden gender "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.527392Z",
     "start_time": "2019-11-21T02:03:42.521601Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Not-in-family', 'Husband', 'Wife', 'Own-child', 'Unmarried',\n",
       "       'Other-relative'], dtype=object)"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['relationship'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.538426Z",
     "start_time": "2019-11-21T02:03:42.529601Z"
    }
   },
   "outputs": [],
   "source": [
    "df['relationship'].replace('Husband', 'Married', inplace=True)\n",
    "df['relationship'].replace('Wife', 'Married', inplace=True)\n",
    "\n",
    "df_t['relationship'].replace('Husband', 'Married', inplace=True)\n",
    "df_t['relationship'].replace('Wife', 'Married', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discard singular values\n",
    "After cleaning up records (see above), some values in a particular column may occur just in the trainning dataset (and not in the test dataset), or vice-versa.\n",
    "\n",
    "That poses a problem later when we vectorize categorical values.\n",
    "So, we will discard these values for simplifications sake."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.555781Z",
     "start_time": "2019-11-21T02:03:42.541932Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19609</th>\n",
       "      <td>32</td>\n",
       "      <td>Private</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Other-relative</td>\n",
       "      <td>0</td>\n",
       "      <td>2205</td>\n",
       "      <td>40</td>\n",
       "      <td>Holand-Netherlands</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age workclass     education marital-status         occupation  \\\n",
       "19609   32   Private  Some-college  Never-married  Machine-op-inspct   \n",
       "\n",
       "         relationship  capital-gain  capital-loss  hours-per-week  \\\n",
       "19609  Other-relative             0          2205              40   \n",
       "\n",
       "           native-country  class  \n",
       "19609  Holand-Netherlands  <=50K  "
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['native-country'] == 'Holand-Netherlands']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.568029Z",
     "start_time": "2019-11-21T02:03:42.558763Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [age, workclass, education, marital-status, occupation, relationship, capital-gain, capital-loss, hours-per-week, native-country, class]\n",
       "Index: []"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_t[df_t['native-country'] == 'Holand-Netherlands']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.575612Z",
     "start_time": "2019-11-21T02:03:42.569808Z"
    }
   },
   "outputs": [],
   "source": [
    "df.drop(19609, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define input features, classification target, and false/positive classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.580712Z",
     "start_time": "2019-11-21T02:03:42.577329Z"
    }
   },
   "outputs": [],
   "source": [
    "target_column = 'class'\n",
    "\n",
    "input_features = df.columns.values.tolist()\n",
    "input_features.remove(target_column)\n",
    "\n",
    "positive_class = '>50K'\n",
    "negative_class = '<=50K'\n",
    "class_labels = [negative_class, positive_class]\n",
    "class_names = [r'Low Income', r'High Income']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filter out data points with missing values (purification) in both datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.632168Z",
     "start_time": "2019-11-21T02:03:42.582356Z"
    }
   },
   "outputs": [],
   "source": [
    "count_before_drop = len(df)\n",
    "\n",
    "df = df.replace(['?'],[None]).dropna(how='any')\n",
    "df_t = df_t.replace(['?'],[None]).dropna(how='any')\n",
    "\n",
    "count_after_drop = len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:42.646706Z",
     "start_time": "2019-11-21T02:03:42.634318Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital-status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "      <th>native-country</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Married</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>11th</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Married</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Married</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         workclass  education      marital-status         occupation  \\\n",
       "0   39         State-gov  Bachelors       Never-married       Adm-clerical   \n",
       "1   50  Self-emp-not-inc  Bachelors  Married-civ-spouse    Exec-managerial   \n",
       "2   38           Private    HS-grad            Divorced  Handlers-cleaners   \n",
       "3   53           Private       11th  Married-civ-spouse  Handlers-cleaners   \n",
       "4   28           Private  Bachelors  Married-civ-spouse     Prof-specialty   \n",
       "\n",
       "    relationship  capital-gain  capital-loss  hours-per-week native-country  \\\n",
       "0  Not-in-family          2174             0              40  United-States   \n",
       "1        Married             0             0              13  United-States   \n",
       "2  Not-in-family             0             0              40  United-States   \n",
       "3        Married             0             0              40  United-States   \n",
       "4        Married             0             0              40           Cuba   \n",
       "\n",
       "   class  \n",
       "0  <=50K  \n",
       "1  <=50K  \n",
       "2  <=50K  \n",
       "3  <=50K  \n",
       "4  <=50K  "
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-04-15T04:47:34.338763",
     "start_time": "2016-04-15T04:47:34.335270"
    }
   },
   "source": [
    "# Split categorical dimensions into multiple boolean dimensions (dummies)\n",
    "\n",
    "Using https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:46.144088Z",
     "start_time": "2019-11-21T02:03:42.648825Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer as DV\n",
    "\n",
    "vectorizer = DV(sparse=False)\n",
    "\n",
    "cat_dicts = df[input_features].T.to_dict().values()\n",
    "vec_cat = vectorizer.fit_transform( cat_dicts )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One line of the table transformed into a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:46.150883Z",
     "start_time": "2019-11-21T02:03:46.145846Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': 39,\n",
       " 'workclass': 'State-gov',\n",
       " 'education': 'Bachelors',\n",
       " 'marital-status': 'Never-married',\n",
       " 'occupation': 'Adm-clerical',\n",
       " 'relationship': 'Not-in-family',\n",
       " 'capital-gain': 2174,\n",
       " 'capital-loss': 0,\n",
       " 'hours-per-week': 40,\n",
       " 'native-country': 'United-States'}"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(cat_dicts.__iter__())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:46.158198Z",
     "start_time": "2019-11-21T02:03:46.152904Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.900e+01, 2.174e+03, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       1.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 4.000e+01, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       1.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 1.000e+00, 0.000e+00,\n",
       "       0.000e+00, 1.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 1.000e+00, 0.000e+00,\n",
       "       0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00, 0.000e+00,\n",
       "       0.000e+00, 1.000e+00, 0.000e+00])"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec_cat[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:47.111527Z",
     "start_time": "2019-11-21T02:03:46.159913Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': 52.0,\n",
       " 'capital-gain': 15024.0,\n",
       " 'education=HS-grad': 1.0,\n",
       " 'hours-per-week': 40.0,\n",
       " 'marital-status=Married-civ-spouse': 1.0,\n",
       " 'native-country=United-States': 1.0,\n",
       " 'occupation=Exec-managerial': 1.0,\n",
       " 'relationship=Married': 1.0,\n",
       " 'workclass=Self-emp-inc': 1.0}"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.inverse_transform(X=vec_cat)[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:47.116815Z",
     "start_time": "2019-11-21T02:03:47.114079Z"
    }
   },
   "outputs": [],
   "source": [
    "input_features_cat = vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.052908Z",
     "start_time": "2019-11-21T02:03:47.119076Z"
    }
   },
   "outputs": [],
   "source": [
    "# Do the same for the test dataset\n",
    "vectorizer_t = DV( sparse = False )\n",
    "cat_test_dicts = df_t[input_features].T.to_dict().values()\n",
    "vec_t_cat = vectorizer_t.fit_transform( cat_test_dicts )\n",
    "input_features_t = vectorizer_t.get_feature_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-04-15T04:49:48.834549",
     "start_time": "2016-04-15T04:49:48.832860"
    }
   },
   "source": [
    "# Split input dataset into training and validation (hold-out) datasets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.073775Z",
     "start_time": "2019-11-21T02:03:49.054573Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "x_input = vec_cat\n",
    "y_input = df[target_column]\n",
    "x_train, x_validation, y_train, y_validation = train_test_split(x_input, y_input, test_size=0.20)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.080947Z",
     "start_time": "2019-11-21T02:03:49.075871Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([41.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  1.,  0.,  0.,  0.,  0., 40.,  1.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,\n",
       "        0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,\n",
       "        0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,\n",
       "        0.,  0.])"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.089866Z",
     "start_time": "2019-11-21T02:03:49.083166Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(24128, 6033, 20.00265243194854)"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x_train), len(x_validation),  len(x_validation)/(len(x_train)+ len(x_validation))*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The test dataset is not split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.103018Z",
     "start_time": "2019-11-21T02:03:49.099025Z"
    }
   },
   "outputs": [],
   "source": [
    "y_test = df_t[target_column]\n",
    "x_test = vec_t_cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check weather the test dataset has the same dimensions as the training dataset\n",
    "\n",
    "If there are attribute-value pairs in the training dataset that do not occur in the test dataset, the notebooks may fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.109478Z",
     "start_time": "2019-11-21T02:03:49.105314Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(set(), set())"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(input_features_cat) - set(input_features_t), set(input_features_t) - set(input_features_cat) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-11-21T02:03:49.115772Z",
     "start_time": "2019-11-21T02:03:49.111162Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all([a==b for a,b in zip(input_features_cat,input_features_t)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}