E03-CrossVal-CreditScoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 03\n",
    "\n",
    "by [Sergio Castelblanco](http://www.linkedin.com/in/sergio-castelblanco/) -- [Jhon Gonzalez](http://linkedin.com/in/jhon-franklin-gonzalez-gamboa-b6123230/)\n",
    "\n",
    "## Data preparation and model evaluation exercise with credit scoring\n",
    "\n",
    "Banks play a crucial role in market economies. They decide who can get finance and on what terms and can make or break investment decisions. For markets and society to function, individuals and companies need access to credit. \n",
    "\n",
    "Credit scoring algorithms, which make a guess at the probability of default, are the method banks use to determine whether or not a loan should be granted. This competition requires participants to improve on the state of the art in credit scoring, by predicting the probability that somebody will experience financial distress in the next two years. [Dataset](https://www.kaggle.com/c/GiveMeSomeCredit)\n",
    "\n",
    "Attribute Information:\n",
    "\n",
    "|Variable Name\t|\tDescription\t|\tType|\n",
    "|----|----|----|\n",
    "|SeriousDlqin2yrs\t|\tPerson experienced 90 days past due delinquency or worse \t|\tY/N|\n",
    "|RevolvingUtilizationOfUnsecuredLines\t|\tTotal balance on credit divided by the sum of credit limits\t|\tpercentage|\n",
    "|age\t|\tAge of borrower in years\t|\tinteger|\n",
    "|NumberOfTime30-59DaysPastDueNotWorse\t|\tNumber of times borrower has been 30-59 days past due |\tinteger|\n",
    "|DebtRatio\t|\tMonthly debt payments\t|\tpercentage|\n",
    "|MonthlyIncome\t|\tMonthly income\t|\treal|\n",
    "|NumberOfOpenCreditLinesAndLoans\t|\tNumber of Open loans |\tinteger|\n",
    "|NumberOfTimes90DaysLate\t|\tNumber of times borrower has been 90 days or more past due.\t|\tinteger|\n",
    "|NumberRealEstateLoansOrLines\t|\tNumber of mortgage and real estate loans\t|\tinteger|\n",
    "|NumberOfTime60-89DaysPastDueNotWorse\t|\tNumber of times borrower has been 60-89 days past due |integer|\n",
    "|NumberOfDependents\t|\tNumber of dependents in family\t|\tinteger|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data into Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>SeriousDlqin2yrs</th>\n",
       "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
       "      <th>age</th>\n",
       "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
       "      <th>DebtRatio</th>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
       "      <th>NumberOfTimes90DaysLate</th>\n",
       "      <th>NumberRealEstateLoansOrLines</th>\n",
       "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
       "      <th>NumberOfDependents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.766127</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.802982</td>\n",
       "      <td>9120.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.957151</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.121876</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.658180</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.085113</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.233810</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.036050</td>\n",
       "      <td>3300.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.907239</td>\n",
       "      <td>49.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.024926</td>\n",
       "      <td>63588.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines   age  \\\n",
       "0           0                 1                              0.766127  45.0   \n",
       "1           1                 0                              0.957151  40.0   \n",
       "2           2                 0                              0.658180  38.0   \n",
       "3           3                 0                              0.233810  30.0   \n",
       "4           4                 0                              0.907239  49.0   \n",
       "\n",
       "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
       "0                                   2.0   0.802982         9120.0   \n",
       "1                                   0.0   0.121876         2600.0   \n",
       "2                                   1.0   0.085113         3042.0   \n",
       "3                                   0.0   0.036050         3300.0   \n",
       "4                                   1.0   0.024926        63588.0   \n",
       "\n",
       "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
       "0                             13.0                      0.0   \n",
       "1                              4.0                      0.0   \n",
       "2                              2.0                      1.0   \n",
       "3                              5.0                      0.0   \n",
       "4                              7.0                      0.0   \n",
       "\n",
       "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
       "0                           6.0                                   0.0   \n",
       "1                           0.0                                   0.0   \n",
       "2                           0.0                                   0.0   \n",
       "3                           0.0                                   0.0   \n",
       "4                           1.0                                   0.0   \n",
       "\n",
       "   NumberOfDependents  \n",
       "0                 2.0  \n",
       "1                 1.0  \n",
       "2                 0.0  \n",
       "3                 0.0  \n",
       "4                 0.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.max_columns', 500)\n",
    "import zipfile\n",
    "with zipfile.ZipFile('../Talleres/KaggleCredit2.csv.zip', 'r') as z:\n",
    "    f = z.open('KaggleCredit2.csv')\n",
    "    data = pd.io.parsers.read_table(f, sep=',')\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = data['SeriousDlqin2yrs']\n",
    "X = data.drop('SeriousDlqin2yrs', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(112915, 11)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3.1\n",
    "\n",
    "Input the missing values of the Age and Number of Dependents "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0                                 0\n",
       "SeriousDlqin2yrs                           0\n",
       "RevolvingUtilizationOfUnsecuredLines       0\n",
       "age                                     4267\n",
       "NumberOfTime30-59DaysPastDueNotWorse       0\n",
       "DebtRatio                                  0\n",
       "MonthlyIncome                              0\n",
       "NumberOfOpenCreditLinesAndLoans            0\n",
       "NumberOfTimes90DaysLate                    0\n",
       "NumberRealEstateLoansOrLines               0\n",
       "NumberOfTime60-89DaysPastDueNotWorse       0\n",
       "NumberOfDependents                      4267\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check for missing values\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(112915, 12)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51.36130439584714"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# mean Age\n",
    "data.age.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.age.fillna(data.age.median(), inplace=True)\n",
    "data.NumberOfDependents.fillna(data.NumberOfDependents.median(), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>SeriousDlqin2yrs</th>\n",
       "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
       "      <th>age</th>\n",
       "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
       "      <th>DebtRatio</th>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
       "      <th>NumberOfTimes90DaysLate</th>\n",
       "      <th>NumberRealEstateLoansOrLines</th>\n",
       "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
       "      <th>NumberOfDependents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.766127</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.802982</td>\n",
       "      <td>9120.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.957151</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.121876</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.658180</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.085113</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.233810</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.036050</td>\n",
       "      <td>3300.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.907239</td>\n",
       "      <td>49.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.024926</td>\n",
       "      <td>63588.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0.213179</td>\n",
       "      <td>74.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.375607</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.754464</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.209940</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0.189169</td>\n",
       "      <td>57.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.606291</td>\n",
       "      <td>23684.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0.644226</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.309476</td>\n",
       "      <td>2500.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0.018798</td>\n",
       "      <td>51.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.531529</td>\n",
       "      <td>6501.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines   age  \\\n",
       "0           0                 1                              0.766127  45.0   \n",
       "1           1                 0                              0.957151  40.0   \n",
       "2           2                 0                              0.658180  38.0   \n",
       "3           3                 0                              0.233810  30.0   \n",
       "4           4                 0                              0.907239  49.0   \n",
       "5           5                 0                              0.213179  74.0   \n",
       "6           6                 0                              0.754464  39.0   \n",
       "7           7                 0                              0.189169  57.0   \n",
       "8           8                 0                              0.644226  30.0   \n",
       "9           9                 0                              0.018798  51.0   \n",
       "\n",
       "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
       "0                                   2.0   0.802982         9120.0   \n",
       "1                                   0.0   0.121876         2600.0   \n",
       "2                                   1.0   0.085113         3042.0   \n",
       "3                                   0.0   0.036050         3300.0   \n",
       "4                                   1.0   0.024926        63588.0   \n",
       "5                                   0.0   0.375607         3500.0   \n",
       "6                                   0.0   0.209940         3500.0   \n",
       "7                                   0.0   0.606291        23684.0   \n",
       "8                                   0.0   0.309476         2500.0   \n",
       "9                                   0.0   0.531529         6501.0   \n",
       "\n",
       "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
       "0                             13.0                      0.0   \n",
       "1                              4.0                      0.0   \n",
       "2                              2.0                      1.0   \n",
       "3                              5.0                      0.0   \n",
       "4                              7.0                      0.0   \n",
       "5                              3.0                      0.0   \n",
       "6                              8.0                      0.0   \n",
       "7                              9.0                      0.0   \n",
       "8                              5.0                      0.0   \n",
       "9                              7.0                      0.0   \n",
       "\n",
       "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
       "0                           6.0                                   0.0   \n",
       "1                           0.0                                   0.0   \n",
       "2                           0.0                                   0.0   \n",
       "3                           0.0                                   0.0   \n",
       "4                           1.0                                   0.0   \n",
       "5                           1.0                                   0.0   \n",
       "6                           0.0                                   0.0   \n",
       "7                           4.0                                   0.0   \n",
       "8                           0.0                                   0.0   \n",
       "9                           2.0                                   0.0   \n",
       "\n",
       "   NumberOfDependents  \n",
       "0                 2.0  \n",
       "1                 1.0  \n",
       "2                 0.0  \n",
       "3                 0.0  \n",
       "4                 0.0  \n",
       "5                 1.0  \n",
       "6                 0.0  \n",
       "7                 2.0  \n",
       "8                 0.0  \n",
       "9                 2.0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0                              0\n",
       "SeriousDlqin2yrs                        0\n",
       "RevolvingUtilizationOfUnsecuredLines    0\n",
       "age                                     0\n",
       "NumberOfTime30-59DaysPastDueNotWorse    0\n",
       "DebtRatio                               0\n",
       "MonthlyIncome                           0\n",
       "NumberOfOpenCreditLinesAndLoans         0\n",
       "NumberOfTimes90DaysLate                 0\n",
       "NumberRealEstateLoansOrLines            0\n",
       "NumberOfTime60-89DaysPastDueNotWorse    0\n",
       "NumberOfDependents                      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3.2\n",
    "\n",
    "From the set of features\n",
    "\n",
    "Select the features that maximize the **F1Score** the model using K-Fold cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "import numpy\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "# load data\n",
    "X = data.drop('SeriousDlqin2yrs', axis=1)\n",
    "y = data.SeriousDlqin2yrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  5.216e+04   7.935e+03   4.836e+03   5.764e+04   5.865e+01   1.448e+06\n",
      "   5.396e+02   8.506e+04   8.778e+01   6.769e+04   3.853e+02]\n",
      "[[  2.000e+00   9.120e+03   0.000e+00   0.000e+00]\n",
      " [  0.000e+00   2.600e+03   0.000e+00   0.000e+00]\n",
      " [  1.000e+00   3.042e+03   1.000e+00   0.000e+00]\n",
      " [  0.000e+00   3.300e+03   0.000e+00   0.000e+00]\n",
      " [  1.000e+00   6.359e+04   0.000e+00   0.000e+00]]\n"
     ]
    }
   ],
   "source": [
    "# Feature Extraction with Univariate Statistical Tests (Chi-squared for classification)\n",
    "import pandas\n",
    "import numpy\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "# load data\n",
    "X = data.drop('SeriousDlqin2yrs', axis=1)\n",
    "y = data.SeriousDlqin2yrs\n",
    "# feature extraction\n",
    "test = SelectKBest(score_func=chi2, k=4)\n",
    "fit = test.fit(X, y)\n",
    "# summarize scores\n",
    "numpy.set_printoptions(precision=3)\n",
    "print(fit.scores_)\n",
    "features = fit.transform(X)\n",
    "# summarize selected features\n",
    "print(features[0:5,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define X and y\n",
    "feature_cols = ['RevolvingUtilizationOfUnsecuredLines', 'age', 'NumberOfTime30-59DaysPastDueNotWorse', 'MonthlyIncome', 'NumberOfTimes90DaysLate', 'NumberOfTime60-89DaysPastDueNotWorse']\n",
    "X = data[feature_cols]\n",
    "y = data.SeriousDlqin2yrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
       "      <th>age</th>\n",
       "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <th>NumberOfTimes90DaysLate</th>\n",
       "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.766127</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9120.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.957151</td>\n",
       "      <td>40.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.658180</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.233810</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.907239</td>\n",
       "      <td>49.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>63588.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.213179</td>\n",
       "      <td>74.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.754464</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.189169</td>\n",
       "      <td>57.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23684.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.644226</td>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2500.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.018798</td>\n",
       "      <td>51.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6501.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   RevolvingUtilizationOfUnsecuredLines   age  \\\n",
       "0                              0.766127  45.0   \n",
       "1                              0.957151  40.0   \n",
       "2                              0.658180  38.0   \n",
       "3                              0.233810  30.0   \n",
       "4                              0.907239  49.0   \n",
       "5                              0.213179  74.0   \n",
       "6                              0.754464  39.0   \n",
       "7                              0.189169  57.0   \n",
       "8                              0.644226  30.0   \n",
       "9                              0.018798  51.0   \n",
       "\n",
       "   NumberOfTime30-59DaysPastDueNotWorse  MonthlyIncome  \\\n",
       "0                                   2.0         9120.0   \n",
       "1                                   0.0         2600.0   \n",
       "2                                   1.0         3042.0   \n",
       "3                                   0.0         3300.0   \n",
       "4                                   1.0        63588.0   \n",
       "5                                   0.0         3500.0   \n",
       "6                                   0.0         3500.0   \n",
       "7                                   0.0        23684.0   \n",
       "8                                   0.0         2500.0   \n",
       "9                                   0.0         6501.0   \n",
       "\n",
       "   NumberOfTimes90DaysLate  NumberOfTime60-89DaysPastDueNotWorse  \n",
       "0                      0.0                                   0.0  \n",
       "1                      0.0                                   0.0  \n",
       "2                      1.0                                   0.0  \n",
       "3                      0.0                                   0.0  \n",
       "4                      0.0                                   0.0  \n",
       "5                      0.0                                   0.0  \n",
       "6                      0.0                                   0.0  \n",
       "7                      0.0                                   0.0  \n",
       "8                      0.0                                   0.0  \n",
       "9                      0.0                                   0.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jfrank/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Runnig the F1 Score using a logistic regresion \n",
    "# train/test split\n",
    "from sklearn.cross_validation import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)\n",
    "\n",
    "# train a logistic regression model\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression(C=1e9)\n",
    "logreg.fit(X_train, y_train)\n",
    "\n",
    "# make predictions for testing set\n",
    "y_pred_class = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[26244,    71],\n",
       "       [ 1813,   101]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_pred_class)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision_score  0.587209302326\n",
      "recall_score     0.0527690700104\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score\n",
    "print('precision_score ', precision_score(y_test, y_pred_class))\n",
    "print('recall_score    ', recall_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f1_score     0.0968360498562\n"
     ]
    }
   ],
   "source": [
    "print('f1_score    ', f1_score(y_test, y_pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3.3\n",
    "\n",
    "Now which is the best set of features selected by AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal number of features : 8\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEbCAYAAAArhqjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VPW9//HXOwn7JmtYwhI2ETeU\nsAgo4orWitZdbN2qrS3V1tZftbe3i7f31i56u9kqbnX3uoutihugIiBBNgHZgkDYEvawZvv8/jgn\nOsaQOUAmM0k+z8djHplz5pyZzwk6n5zv8vnKzHDOOeeqk5bsAJxzzqU+TxbOOefi8mThnHMuLk8W\nzjnn4vJk4ZxzLi5PFs455+LyZOGccy6ujCgHSeoEjAS6AnuBT4BcMytPYGzOOedShKqblCdpDHA7\n0A6YCxQATYH+QB/geeBuM9uZ+FCdc84lS7xk8Qfgr2a2porXMoDzgHQzeyFxITrnnEu2apOFc845\nBxE7uCXdIqm1Ag9J+ljSWYkOzjnnXGqIOhrqurBf4iygI3AtcFfConLOOZdSoiYLhT/PBR4xs/kx\n+5xzztVzUZPFHElvEiSLyZJaAT5s1jnnGohIHdyS0oBBQJ6ZbZfUHuhmZgsSHaBzzrnkizQpz8zK\nJW0CBoZDZp1zzjUgUWdw/w64DFgMlIW7DXgvQXE555xLIVGboZYCx5nZ/sSH5JxzLtVE7eDOAxol\nMhDnnHOpK2r/wx5gnqR3gM/vLszs5oRE5ZxzLqVETRaTwodzzrkGKHJtKEmNCarNAiw1s5KEReWc\ncy6lRO3gPhV4FPiMYOZ2d+BqM/PRUM451wBETRZzgCvNbGm43R942swGJzg+55xzKSDqaKhGFYkC\nwMyW4aOjnHOuwYjawZ0r6SHg8XB7PDAnMSEdmg4dOlivXr2SHYZzztUpc+bM2WxmHeMdFzVZ3AR8\nH7iZoM/iPeDvhx5ezevVqxe5ubnJDsM55+oUSaujHBe1NtR+4J7w4ZxzroGpNllIetbMLpW0kKAW\n1JeY2XEJi8w551zKiHdncUv487xEB+Kccy51VTsaysw2hE+/Z2arYx/A9xIfnnPOuVQQdejsmVXs\nO6cmA3HOOZe6qk0Wkm4K+yuOlLQg5rEKiLtKnqSxkpZKWiHp9ipe/19J88LHMknbY167WtLy8HH1\noVycc865mhGvz+Ip4HXgt0Dsl32RmW2t7kRJ6cC9BHcl+cBsSZPMbHHFMWb2o5jjfwCcED5vB/wS\nyCHoWJ8Tnrst6oU555yrOdUmCzPbAewArgCQ1AloCrSU1NLM1lRz+lBghZnlhec+A4wjWG2vKlcQ\nJAiAs4G3KhKSpLeAscDTUS7KOZc8W3cXM2neOtLTRFbb5nRv14ysts1p2ig92aG5wxB1WdWvE8yx\n6AoUAD2BJcDR1ZzWDVgbs50PDDvA+/cEsoF3qzm3W5RYnXPJsWxTEY9MX8WLH69jf2n5V17v0LIJ\n3ds1o3vb5mS1bUb3ds0/f971iGY0zojaheqSIeoM7t8Aw4G3zewESWMI7zaqoSr2Hahq4eXA82ZW\nsb53pHMl3QjcCNCjR4844Tjnalp5uTFteSEPf7CK95dvpklGGt84MYtrRvTiiOaNyN+2h7Vb97J2\n6x7yt+1l7bY9zF27jX8v3EBZ+Rf/S6cJOrduSla7MJG0bU73dl8klc6tm5KeVtXXgqstUZNFiZlt\nkZQmKc3Mpkj6XZxz8glKmVfIAtYf4NjLCcqJxJ57aqVzp1Y+ycwmAhMBcnJyoi3M4Zw7bHuLy3jh\n43wemb6KlYW76dSqCbedfSRXDO1BuxaNPz8us3VTBvf86vmlZeVs3LmPtVv3Bgll217yt+5h7bY9\nzFi5hZd2riO2IHZGmuh6RLOv3JlUNHN1bNkEyZNJIkVNFtsltSSoCfWkpAKgNM45s4F+krKBdQQJ\n4crKB0k6EmgLzIjZPRn4H0ltw+2zgDsixuqcS5ANO/by2IzVPDVrDTv2lnBstzb86bJBnHtsl4Nq\nRspITyOrbfBlD+2/8vr+0jLWb9/3xZ3JtvDOZOse3l6yic27ir90fJOMtM8TyFFdWvONE7rRL7PV\n4V6uixE1WYwD9gI/Iqg42wa4s7oTzKxU0gSCL/504GEzWyTpTiDXzCqWab0CeMZiFtYws62S/osg\n4QDcGW/0lXMuceat3c7DH6zitYUbKDfj7KM7c92obHJ6tk3IX/RNMtLJ7tCC7A4tqnx9b3FZeEfy\nRRKpSCofLM/jH1NXMqj7EVya053zju9C66a+osLhirr4UTawwcz2hdvNgEwz+yyx4UWXk5NjXnXW\nuZpTWlbO5EWbeHj6Kuas3karJhlcNqQ7V4/oRfd2zZMd3gFt3rWfl+eu49nctSzbtIumjdI455gu\nXJKTxfDs9qR538eXSJpjZjlxj4uYLHKBEWZWHG43Bqab2ZDDjrSGeLJwrmbs2FvC/81ew6Mfrmbd\n9r30aNeca0f24pKc7rRsErUxIvnMjAX5O3g2dy2T5q+naF8p3ds14+ITu3PR4G5hE5ir6WQxz8wG\nVdo338yOP4wYa5QnC+cOz6rNu/nn9FU8NyefPcVlDO/djutGZnP6UZl1fiTSvpIyJi/ayLO5a5m+\nYgsSjOrbgYsHZ3H20Z0b9ByQqMki6p8JhZLOr+hnkDQO2Hw4ATrnks/MmLFyCw9PX8U7nxbQKC2N\nrx/flWtH9uKYbm2SHV6NadoonXGDujFuUDfWbt3DCx/n81xuPrc8M4/WTTMYN6gbl+RkcWy3Nj6q\n6gCi3ln0AZ4kmJQngglz3zKzFYkNLzq/s3Auun0lZUyav56HP1jFpxuLaN+iMeOH9+Sq4T3o1Kpp\nssOrFeXlxsy8LTybu5bXP9nI/tJyBnRuxSU53blgUFfat2yS7BBrRY02Q8W8acvwnKLDCS4RPFk4\nF19h0X6emLmaJ2etZvOuYgZ0bsV1I7M5f1DXBt0Us2NvCf9asJ5nc/OZv3Y7jdLF6QMyuXRIFqf0\n60hGev2dXV4jyULSVWb2hKRbq3rdzFJmmVVPFs4d2OL1O3l4+iomzVtPcVk5pw/oxHWjshnRp703\nu1SybFMRz+Wu5cWP17FldzGdWjXhGydmcUlOFn06tkx2eDWupvosKoYL+OwW5+qY8nLjnU8LePiD\nVczI20KzRulcPrQ714zoRe96+KVXU/pntuI/vjaQ/zd2AO9+WsBzuWt54P087pu2ksE923JpThZf\nO65rnRoZVhPi3Vn8zsx+KukSM3uuFuM6aH5n4dwX3l9eyG9f+5TFG3bStU1Trh7Ri8uH9KBNc5+c\ndigKivbx0sfB3I2Vhbtp1iidc4/twqU5WQzNblen785qqhlqIXAiMMvMTqzB+GqcJwvnYNH6Hdz1\n+qe8v3wzWW2b8eOz+vP147rW6zb32mRmfLxmO8/PWcur8zewa38pvdo35+LBWVw0OIsubZolO8SD\nVlPJ4g8EVV1bAHtiXwLMzFofbqA1xZOFa8jyt+3hnjeX8dK8dbRp1ogJY/ryzZN60iSj4XZaJ9qe\n4lJeX7iR5+asZWbeVtLTxD+vHcLJ/TomO7SDUtOT8l4xs3E1ElmCeLJwDdGOPSX8feoKHvnwMwCu\nG5nNTaf2oU0zb26qTau37Gb8g7Po1KoJL9w0ok41S9XopLxUTxTONTT7S8t4fMZq/vruCnbuK+Eb\nJ2Rx61n96XZE3WsGqQ96tm/Bjaf05hevLGL2Z9sYmt0u2SHVuGqThaQPzGyUpCKCxYdi02VKNUM5\n1xCUlxuT5q/nj28uJX/bXk7p35Hbxw5gYFf/XzHZLhncnT+9vZz7pq1seMnCzEaFP33orHNJNn3F\nZv7ntSUsWr+To7u25q5vHMeofh2SHZYLNWuczjUjenHPW8v4dONOBnSuXwk80hAJSX0kNQmfnyrp\nZklHJDY05xzAkg07ufrhjxj/4Cy27ynhT5cN4tUJozxRpKBvndST5o3TuX9aXrJDqXFRx9O9AJRJ\n6gs8BGQDTyUsKucc67fv5SfPzefcv7zP3DXb+I9zj+KdH4/mghO6+ZoMKeqI5o25YmgPJs1fT/62\nPfFPqEOiJotyMysFLgT+ZGY/ArokLiznGq4de0u46/VPGfPHqUyat54bTu7Ne/9vDDec0rtB12+q\nK64flY2AB99flexQalTU+eolkq4Arga+Hu7zsXnO1aD9pWU8MXMNf313Odv3lHDhCd348Vn9fZGe\nOqbrEc0YN6gbz8xew82n96Ndi8bJDqlGRL2zuBY4CfhvM1sVLrP6RLyTJI2VtFTSCkm3H+CYSyUt\nlrRI0lMx+38f7lsi6S+qSwOXnTsIFSOczrhnGv/1r8Uc07UN//rBKP73skGeKOqo747uzb6Sch4N\n57/UB1HnWSwGbgaQ1BZoZWZ3VXeOpHTgXuBMIB+YLWlS+F4Vx/QD7gBGmtk2SZ3C/SOAkcBx4aEf\nAKOBqdEvzbnU9+HKzdz1+qcsyN/BgM6teOy6oZzSv27NAHZf1S+zFWcclcmjMz7jO6N707xx3S86\nGHU01FRJrSW1A+YDj0iKV558KLDCzPLCtbufASpP7rsBuNfMtgGYWUG434CmQGOgCUGT16YosTpX\nFyzdWMS1j3zElQ/MYnPRfu6+5Hj+ffPJnijqkZtO7c32PSU889HaZIdSI6KmuzZmtlPSt4FHzOyX\nkhbEOacbwYp6FfKBYZWO6Q8gaTqQDvzKzN4wsxmSpgAbCCYC/s3MlkSM1bmUtXHHPu55aynPz8mn\nRZMMbj9nANeM6OUd1/XQ4J7tGNKrLQ99sIpvntSTRnW8mGPUZJEhqQtwKfAfEc+pqo+hciGqDKAf\ncCqQBbwv6RigA3BUuA/gLUmnmNl7X/oA6UaCQof06NEjYljO1b6d+0q4f9pKHvpgFeXlcO3IbCaM\n6UvbetL56ap206l9uO6fubw6fz3fODEr/gkpLGqyuBOYDHxgZrMl9QaWxzknH+ges50FrK/imJlm\nVgKskrSUL5LHTDPbBSDpdWA48KVkYWYTgYkQFBKMeC3OJVx5uZG/bS/LNhXxyfodPDZjNVt3FzNu\nUFd+ctaRdG/nHdcNwZgjO3FkZivum7aSCwbV7fkxUTu4nwOei9nOAy6Kc9psoF84cmodcDlwZaVj\nXgauAP4pqQNBs1Qe0Bu4QdJvCe5QRgN/ihKrc7WpvNxYt30vywuKWLZpF8s2FbF80y5WFOxib0nZ\n58eN6NOeO845imOz2iQxWlfbJPGd0b259dn5TFlawOlHZSY7pEMWKVlIagpcDxxN0PEMgJldd6Bz\nzKxU0gSCO5J04GEzWyTpTiDXzCaFr50laTFQBtxmZlskPQ+cBiwkaLp6w8xePaQrdK4GmBkbduz7\nPBks21TEsk1FrCjYxe7iL5JCp1ZN6J/ZisuHdqd/Ziv6Z7akb6dWXjK8Afv68V25+81l3DdtZZ1O\nFlHXs3gO+JTgzuBOYDywxMxuSWx40fl6Fq4mmBmbdu7/PBks37SLZQVFrNi0i6L9pZ8f16FlE/pn\ntqR/Ziv6Vfzs1JIjmnsfhPuqR6av4tevLuaFm05icM/Uqkhbo+tZAH3N7BJJ48zs0XDy3OTDC9G5\n5DEzCov2f9F0FNOMVLTvi6TQvkVj+mW25MITu9EvsxX9OwWJwTum3cG4bEh3/vLOcv4xNY8Hr06t\nZBFV5HIf4c/t4WiljUCvhETkXAJs213MqwvW8+nGIpZvChLDjr0ln7/etnkj+mW2YtygruFdQtCE\n1L5lkyRG7eqL5o0z+NZJvfjzO8tZtqmI/pl1b9WHqMliYjhz+z+BSUBL4BcJi8q5GrJ7fykPfbCK\nB97Lo2h/KW2aNaJ/Zku+dlyXz+8S+mW2okPLxnVqKUxX91w9ohcT38vj/ml53H3p8ckO56BFHQ31\nYPh0GsFIJedS2v7SMp6atYZ7p6xg865izj46kx+d2Z8jM1t5UnBJ0a5FYy4b0p0nZq7mx2f1p2sd\nWwI33rKqt1b3upnFK/nhXK0qKzdemruO/31rGeu272VEn/Y88K0jOaFH22SH5hzfPjmbx2eu5qEP\nVvGf5w1MdjgHJd6dRd1rWHMNkpkxedEm7n5zKcsLdnFcVht+d5EvO+pSS1bb5px/fFee/mgNPzit\nb50aPRdvDe5f11Ygzh2qD1ds5neTlzJ/7Xb6dGzBP8afyNhjOntzk0tJ3xndm5fmruOxGau5+fR+\nyQ4nsqhVZx+NXXNbUltJDycuLOfiW5C/nW8+NIsrH5xF4c59/P6i45j8w1M459gunihcyhrQuTWn\nDejEPz/8jL0xEzpTXdTRUMeZ2faKjXDtiRMSFJNz1VpRsIu731zK659spF2LxvzneQMZP6yHV251\ndcZ3R/fh0vtn8NyctXzrpF7JDieSqMkiTVLbinUnwnUt6v5qHq5OWbd9L39+exnPz8mnWaN0fnhG\nP64flU2rpl5Kw9UtQ3q15cQeRzDxvTyuHNqDjDpQvjzqF/7dwIdhzSYjKFX+3wmLyrkYW3bt5+9T\nV/L4zNVgQXnv753axyfMuTpLEjed2pcbHsvl3ws3MG5Qt2SHFFfUeRaPScolKO4n4Buxy6M6lwhF\n+0p48P1VPPh+HntLyrh4cBa3nNGfbnVsfLpzVTl9QCf6dWrJfdPyOP/4rinfzxa5KSlMDp4gXMLt\nKynjiZmr+fvUlWzdXcw5x3Tmx2f1p28nH8nt6o+0NHHjKb257fkFTFtWyKlHdkp2SNXyfgeXMkrL\nynnx43X86e1lrN+xj1F9O3Db2UdyfPcj4p/sXB00blA37nlrGf+YutKThXPxmBlvfLKRP765lJWF\nuzm++xH88ZLjGdHXJ9S5+q1xRhrXj8rmN/9ewtw121K60kDUeRa/i7LPuYP1wfLNjLt3Ojc9+TGS\nuO+qwbz8vRGeKFyDccXQHrRp1oj7pq1MdijVijpe68wq9p1Tk4G4hmXe2u1c+cBMrnpoFlt2FfOH\ni4MJdT7z2jU0LZpk8K2TevLm4k2sKNiV7HAOKF4hwZuA7wF9JC2IeakV8GEiA3P102ebd/Pb15cw\nedEm2rdozC+/PpArh/WgSYZPqHMNV0X58onvreT3F6dm+fJ4fRZPAa8DvwVuj9lfZGZbExaVq5fK\ny43rHp1Nwc793Hpmf64blU3LJt5t5lyHlk24bEh3nv5oDbeeeSSd2zRNdkhfUW0zlJntMLPPgD8D\nW81stZmtBkokDYv35pLGSloqaYWk2w9wzKWSFktaFC7XWrG/h6Q3JS0JX+91MBfmUs+HK7eQV7ib\n31xwDDef3s8ThXMxbji5N+UGD09flexQqhS1z+IfQGxj2u5w3wFJSgfuJejbGAhcIWlgpWP6AXcA\nI83saOCHMS8/BvzBzI4ChgIFEWN1KerxmZ/RrkVjzjm2c7JDcS7ldG/XnK8d24UnZ65mx56S+CfU\nsqjJQmZmFRtmVk78JqyhwAozyzOzYuAZYFylY24A7q2oOWVmBQBhUskws7fC/bvMbE/EWF0K2rhj\nH28vKeDSnO7eP+HcAXxndG92F5fxxKzVyQ7lK6ImizxJN0tqFD5uAfLinNMNWBuznR/ui9Uf6C9p\nuqSZksbG7N8u6UVJcyX9IbxT+RJJN0rKlZRbWFgY8VJcMjz90RrKzRg/rEeyQ3EuZR3dtQ2j+3fk\nkemr2FeSWuXLoyaL7wIjgHUEX/rDgBvjnFPV+EertJ0B9ANOBa4AHgzXzcgATgZ+AgwhWPf7mq+8\nmdlEM8sxs5yOHTtGvBRX20rKynlm9hpG9+9I93bNkx2Ocyntu6P7sHlXMc/PyU92KF8SKVmYWYGZ\nXW5mncws08yurGgyqkY+0D1mOwtYX8Uxr5hZiZmtApYSJI98YG7YhFUKvAycGCVWl3reXryJTTv3\nc9WwnskOxbmUN7x3O47vHpQvLy0rT3Y4n4s6g7u/pHckfRJuHyfp53FOmw30k5QtqTFwOTCp0jEv\nA2PC9+xA0PyUF57bVlLF7cJpeBHDOuuJWavpdkQzxgxI7do3zqUCSdw0ujdrtu7h9U82Jjucz0Vt\nhnqAYNRSCYCZLSD48j+g8I5gAjAZWAI8a2aLJN0p6fzwsMnAFkmLgSnAbWa2xczKCJqg3pG0kKBJ\n64GDuzSXCvIKdzF9xRauHNaD9DSfme1cFGcN7Ezvji24b9pKYsYWJVXUge7NzeyjSmUYSuOdZGav\nAa9V2veLmOcG3Bo+Kp/7FnBcxPhcinpy1hoapYtLc7rHP9g5BwTly79zSm9++sJCPlixmZP7Jb9P\nNuqdxWZJfQg7qCVdDGxIWFSuXthbXMbzc/I5++jOdGzlq9o5dzAuOKEbma2bpEyBwajJ4vvA/cAA\nSesIJs99N2FRuXrh1QXr2bG3hKuGe8e2cwerSUY6143MZvqKLSzI357scOInC0lpQI6ZnQF0BAaY\n2aiw7IdzB/TkzNX069SSYdntkh2Kc3XSlcN60KppRkrcXcRNFuFs7Qnh891mVpTwqFydtzB/B/Pz\nd3DV8J5ecty5Q9SqaSO+Obwnr3+ykVWbdyc1lqjNUG9J+omk7pLaVTwSGpmr056YuZpmjdK58MTK\nk/adcwfj2pHZNEpPY+J78YpmJFbUZHEdQb/Fe8Cc8JGbqKBc3bZjbwmvzF/HBSd0pXXTRskOx7k6\nrWOrJlw8OIsX5uRTsHNf0uKI2mdxlZllV3r0roX4XB30wpx89pWUM95nbDtXI248uTel5eU8PP2z\npMUQtc/ij7UQi6sHzIwnZ63mhB5HcEy3NskOx7l6oVeHFpwTli/fuS855cujNkO9KekieU+li2NG\n3hZWFu72OlDO1bCbRvehaH8pT81ak5TPj5osbgWeA4ol7ZRUJGlnAuNyddSTM9dwRPNGfO24LskO\nxbl65ZhubRjVtwMPfZCc8uVRq862MrM0M2tkZq3D7daJDs7VLQU79zF50UYuGZxF00a+wJFzNe2m\nU/tQWLSfl+auq/XPjnpngaTzJf0xfJyXyKBc3fTM7LWUlhtXehOUcwkxok97ju3Whonv5VFWXrsF\nBqOWKL8LuIWgTPhi4JZwn3MAlJaV8/RHazi5XweyO7RIdjjO1UuS+O7oPqzavJs3F9Vu+fKodxbn\nAmea2cNm9jAwNtznHADvflrAhh37vA6Ucwk29pjO9GrfnH/UcvnyyM1QwBExz31MpPuSJ2atoUub\nppzuCxw5l1DpaeLGU/qwIH8HM1ZuqbXPjZosfgvMlfRPSY8SzOD+n8SF5eqSzzbv5r1lhVw+pAcZ\n6Qfz94dz7lB848RudGjZhH/UYoHBqKOhngaGAy+Gj5PM7JlEBubqjqc+WkN6mrh8qC9w5FxtaNoo\nnetG9eL95Zv5ZN2OWvnMqB3cFwJ7zGySmb0C7JN0QWJDc3XBvpIynstdy9lHZ5LZummyw3GuwRg/\nrCctm9Re+fKobQa/NLPP05eZbQd+Ge8kSWMlLZW0QtLtBzjmUkmLJS2S9FSl11pLWifpbxHjdLXs\ntYUb2LanxGdsO1fL2jRrxPjhPXht4QZWb0l8+fKoyaKq46pdv1tSOnAvcA4wELhC0sBKx/QD7gBG\nmtnRBCvwxfovYFrEGF0SPDFzNb07tuCkPu2THYpzDc71I7PJSEvjgfcTX748arLIlXSPpD6Sekv6\nX4JO7uoMBVaYWZ6ZFQPPAOMqHXMDcK+ZbQMws4KKFyQNBjKBNyPG6GrZovU7+HjNdsYP8wWOnEuG\nTq2bctHgLLbvKUn4MNpq7w5i/AD4T+D/wu03gZ/HOacbsDZmOx8YVumY/gCSpgPpwK/M7I2wLPrd\nwDeB0yPG6GrZEzPX0LRRGhefmJXsUJxrsH5zwTGkpyX+j7VIycLMdgNV9jlUo6roK6e+DKAfcCqQ\nBbwv6RjgKuA1M1tb3V+skm4EbgTo0aPHQYbnDkfRvhJembeO84/vSpvmvsCRc8lSG4kCot9ZHIp8\nIHYsZRawvopjZppZCbBK0lKC5HEScLKk7wEtgcaSdpnZlxKWmU0EJgLk5OTUbqGUBu6luevYU1zm\nM7adayASOYNqNtBPUrakxsDlwKRKx7wMjAGQ1IGgWSrPzMabWQ8z6wX8BHiscqJwyWNmPD5jNcdl\nteG4rCPin+Ccq/OqTRaSfhf+vORg39jMSoEJwGRgCfCsmS2SdKek88PDJgNbJC0GpgC3mVntzV93\nh+SjVVtZXrDLh8s614Couh50SQuBE4FZZnZirUV1CHJyciw3NzfZYTQIP3h6LtOWFjDrZ2fQrLGv\nW+FcXSZpjpnlxDsuXp/FG8BmoEW4Mp4IOqkFmC+A1PAUFu3njU828M3hvTxRONeAVNsMZWa3mVkb\n4N+xK+T5SnkN17O5aykpM8YP99FnzjUkUYfOjpOUCQwJd80ys8LEheVSUVm58dSsNYzo054+HVsm\nOxznXC2KWkjwEuAj4BLgUuAjSRcnMjCXeqYuLWDd9r0+XNa5BijqPIufA0MqynFI6gi8DTyfqMBc\n6nli5mo6tWrCmQMzkx2Kc66WRS4kGFu3CdhyEOe6emDt1j1MXVbI5UN70MgXOHKuwYl6Z/GGpMnA\n0+H2ZcBriQnJpaInZ60hTeIKX+DIuQYpagf3bZK+AYwiGDY70cxeSmhkLmXsLy3j2dy1nD6gE13a\nNEt2OM65JIhcG8rMKpZUdQ3MG59sZOvuYu/Ydq4B88ZnF9cTM1fTs31zRvXtkOxQnHNJ4snCVevT\njTuZ/dk2xg/rQVotlUJ2zqWeyM1QYeXYAQTlPpaGq9+5eu6JmatpnJHGJYO9Y9u5hixSspD0NeA+\nYCVBB3e2pO+Y2euJDM4l1679pbz08TrOO64LbVs0TnY4zrkkinpncTcwxsxWAEjqA/wb8GRRj708\ndx27fYEj5xzR+ywKKhJFKA8oONDBru4zM56YuZqBXVpzQndf4Mi5hq7aO4twbgXAIkmvAc8S9Flc\nQrASnqunPl6zjU83FvE/Fx5LdeugO+cahnjNUF+Peb4JGB0+LwTaJiQilxIen7GaVk0yGDeoa7JD\ncc6lgGqThZldW1uBuNSxZdd+Xlu4kSuGdqdFk8gD5pxz9VjUEuUdJf1M0kRJD1c8Ipw3VtJSSSsk\n3X6AYy6VtFjSIklPhfsGSZouIwAEAAAcOElEQVQR7lsg6bKDuyx3OJ6bk09xWTnjvWPbOReK+mfj\nK8D7BGXJy6KcICkduBc4E8gHZkuaZGaLY47pB9wBjDSzbZI6hS/tAb5lZssldQXmSJpsZtsjxusO\nUXm58eSs1QzNbkf/zFbJDsc5lyKiJovmZvbTg3zvocAKM8sDkPQMMA5YHHPMDcC9ZrYNoKIMupkt\nqzjAzNZLKgA6Ap4sEmza8kLWbt3LbWcPSHYozrkUEnXo7L8knXuQ790NWBuznR/ui9Uf6C9puqSZ\nksZWfhNJQ4HGBBMCXYI9OXM1HVo2ZuzRnZMdinMuhURNFrcQJIy9knZKKpK0M845VY23tErbGUA/\n4FTgCuBBSZ8P6pfUBXgcuNbMyr/yAdKNknIl5RYW+pLgh2vd9r28+2kBlw3pTuMMLxvmnPtCpG8E\nM2tlZmlm1szMWofbreOclg/EFhTKAtZXccwrZlZiZquApQTJA0mtCWaJ/9zMZh4grolmlmNmOR07\ndoxyKa4aT89agwFXDO2R7FCccymm2mQhqVec1yUp6wAvzwb6ScoOixBeDkyqdMzLwJjwvToQNEvl\nhce/BDxmZs/Fuwh3+IpLy3lm9lpOO7ITWW2bJzsc51yKidfB/QdJaQSjoeYQTMZrCvQl+JI/Hfgl\nwR3Cl5hZqaQJwGQgHXjYzBZJuhPINbNJ4WtnSVpMMMrqNjPbIukq4BSgvaRrwre8xszmHd7lugOZ\nvGgjm3ft9zpQzrkqyaxyN0KlA6SBwHhgJNCFYFjrEoI1uJ83s32JDjKKnJwcy83NTXYYddZl989g\n/Y69TP3JGNJ93QrnGgxJc8wsJ95xcYfOhvMi/qNGonIpafmmImat2spPxw7wROGcq5IPeXE8OWsN\njdPTuDTnQN1PzrmGzpNFA7enuJQX5uRzzrGdad+ySbLDcc6lKE8WDdwr89ZTtL/UO7adc9WKWkhQ\nkq6S9Itwu0c4s9rVYRULHA3o3Iqcnl5x3jl3YFHvLP4OnEQwyxqgiKBIoKvD5q3dzqL1Oxk/vKcv\ncOScq1bUQoLDzOxESXMBwgqxjRMYl6sFT8xcQ4vG6Vx4QuWSXc4592VR7yxKwpLjBsH6FsBXajW5\numPb7mJeXbCeC07oRktf4Mg5F0fUZPEXgvIbnST9N/AB8D8Ji8ol3PNz8ikuLfeObedcJJH+pDSz\nJyXNISjvIeACM1uS0MhcwhSXlvPkrNXk9GzLUV3i1YN0zrkIySKsDbXAzI4BPk18SC4RduwtYerS\nAt5eUsDUpQUU7Svl1rOOTHZYzrk6Ikq5j3JJ8yX1MLM1tRGUqxlrt+7h7SWbeHvJJmblbaW03Gjf\nojHnHNOZscd0ZsyRneK/iXPOEX00VBdgkaSPgN0VO83s/IRE5Q5JebmxcN0O3l6yibcWb+LTjUUA\n9O3Ukm+f3JszB3ZiUPe2Xv/JOXfQoiaLXyc0CnfI9pWUMWPlFt5asol3lmxi0879pAlyerXjP849\nijMGZpLdoUWyw3TO1XFRO7inScoEhoS7PjKzgsSF5aqzdXcx735awNuLN/He8kL2FJfRvHE6o/t3\n5IyjMjltQCfatvBpMM65mhMpWUi6FPgDMJVgNNRfJd1mZs8nMDYXI69wV9D/sLiA3NVbKTfIbN2E\nC0/oxpkDMxneuz1NG6UnO0znXD0VtRnqP4AhFXcT4aS8twFPFglSVm7MXbONt5Zs4u3Fm1hZGHQV\nHdWlNRPG9OXMgZ05pltrL9PhnKsVUZNFWqVmpy14xdoat6e4lPeXb+btxZt499MCtuwuJiNNDO/d\nnm8O78kZAzN9fWznXFJETRZvSJoMPB1uXwa8Hu8kSWOBPxOswf2gmd1VxTGXAr8iKCUy38yuDPdf\nDfw8POw3ZvZoxFjrlIKd+3gn7H/4YMVm9peW06ppBqcN6MQZR2Uy+siOtG7aKNlhOucauKgd3LdJ\n+gYwiqDPYqKZvVTdOWEtqXuBM4F8YLakSeEyrRXH9APuAEaGxQk7hfvbAb8EcgiSyJzw3G0HfYUp\n7C/vLOeet5YBkNW2GVcO68GZR2UyJLsdjdL9xs05lzqidnBnA6+Z2YvhdjNJvczss2pOGwqsMLO8\n8JxngHHA4phjbgDurUgCMU1dZwNvmdnW8Ny3gLF8cWdT5xUU7eNvU1Zw2oBO/L+xR3JkZivvf3DO\npayof74+x5erzJaF+6rTDVgbs50f7ovVH+gvabqkmWGzVdRz67QH319FaVk5vzhvIAM6e0e1cy61\nRe2zyDCz4ooNMyuOsJ5FVd9+VsXn9wNOBbKA9yUdE/FcJN0I3AjQo0ePOOGkjm27i3li5mq+fnxX\nevmEOedcHRD1zqJQ0uelPSSNAzbHOScf6B6znQWsr+KYV8ysxMxWAUsJkkeUczGziWaWY2Y5HTt2\njHgpyffI9FXsKS7j+2P6JjsU55yLJGqy+C7wM0lrJK0Ffgp8J845s4F+krLDu5DLgUmVjnkZGAMg\nqQNBs1QeMBk4S1JbSW2Bs8J9dd7OfSX888PPOPvoTPpntkp2OM45F0nU0VArgeGSWgIys6II55RK\nmkDwJZ8OPGxmiyTdCeSa2SS+SAqLCfpBbjOzLQCS/osg4QDcWdHZXdc9PmM1O/eVMmFMv2SH4pxz\nkcnsK10BXz1IugV4BCgCHgBOBG43szcTG150OTk5lpubm+wwqrWnuJRRv5vCsd3a8Oh1Q5MdjnPO\nIWmOmeXEOy5qM9R1ZraToDmoE3At8JUJdq56T3+0lq27i/nBad5X4ZyrW6Imi4rRSecCj5jZfKoe\nseQOYH9pGRPfW8mw7Hbk9GqX7HCcc+6gRE0WcyS9SZAsJktqxZfnXbg4np+Tz6ad+/nBad5X4Zyr\ne6LOs7geGATkmdkeSe0JmqJcBCVl5fxj6kqO734EI/u2T3Y4zjl30KKOhioHPo7Z3kJQedZFMGne\nevK37eVXXz/aZ2o75+okr1aXYOXlxt+nrmBA51acflSnZIfjnHOHxJNFgr2xaCMrC3cz4bS+flfh\nnKuzovZZVJQcz4w9x8zWJCKo+sLM+Ou7K+jdsQXnHNMl2eE459whi1qi/AcE60ts4otRUAYcl6C4\n6oUpSwtYsmEnf7zkeNLT/K7COVd3Rb2zuAU4sqIUh4uv4q4iq20zxg3qmuxwnHPusETts1gL7Ehk\nIPXNjJVbmLtmO98d3cdXvXPO1XlR7yzygKmS/g3sr9hpZvckJKp64K/vrqBTqyZcPDgr2aE459xh\ni5os1oSPxuHDVWPO6q3MyNvCz792FE0bpSc7HOecO2xRJ+X9GiAs82FmtiuhUdVxf3t3Be1aNObK\nYXVn9T7nnKtOpMZ0ScdImgt8AiySNEfS0YkNrW76ZN0Opiwt5PpR2TRvHHlksnPOpbSoPa8TgVvN\nrKeZ9QR+TLCuhavk3ikraNU0g2+e1DPZoTjnXI2JmixamNmUig0zmwq0SEhEddjyTUW8sWgjV5/U\ni9ZNGyU7HOecqzGRR0NJ+k/g8XD7KmBVYkKqu/4+dSVNM9K5blR2skNxzrkaFXmlPKAj8CLwUvg8\nbolySWMlLZW0QtLtVbx+jaRCSfPCx7djXvu9pEWSlkj6i1K8sNLqLbuZNH8944f1oF0LHzDmnKtf\noo6G2gbcfDBvHNaSuhc4E8gHZkuaZGaLKx36f2Y2odK5I4CRfFFO5ANgNDD1YGKoTfdNW0l6mrjh\nlN7JDsU552pctclC0p/M7IeSXiWoBfUlZnZ+NacPBVaYWV74Xs8A44DKyaIqBjQlmNMhoBFBXaqU\ntGHHXp6fk89lQ7qT2bppssNxzrkaF+/OoqKP4o+H8N7dCMqEVMgHhlVx3EWSTgGWAT8ys7VmNkPS\nFGADQbL4m5ktOYQYasX90/Iwg++c0ifZoTjnXEJU22dhZnPCp4PMbFrsg2CZ1epU1cdQ+e7kVaCX\nmR0HvA08CiCpL3AUkEWQdE4LE8qXP0C6UVKupNzCwsI44SRGYdF+npm9hgtO6Eb3ds2TEoNzziVa\n1A7uq6vYd02cc/KB7jHbWcD62APMbIuZVdSaegAYHD6/EJhpZrvC2eKvA8Mrf4CZTTSzHDPL6dix\nY/yrSICHPljF/tJyvneq31U45+qvapOFpCvC/opsSZNiHlOIvwb3bKCfpGxJjYHLgUmV3j92RaDz\ngYqmpjXAaEkZkhoRdG6nXDPU9j3FPD7jM752bBd6d2yZ7HCccy5h4vVZfEjQb9ABuDtmfxGwoLoT\nzaxU0gRgMpAOPGxmiyTdCeSa2STgZknnA6XAVr64W3keOA1YSNB09YaZvXowF1Yb/vnhZ+wuLuP7\nY/omOxTnnEsomX1lkFOdlJOTY7m5ubX2ebv2lzLyrncZ0qsdD16dU2uf65xzNUnSHDOL+yUWtZDg\ncEmzJe2SVCypTNLOww+z7npi5mp27C1hwml+V+Gcq/+idnD/DbgCWA40A74N/DVRQaW6fSVlPPh+\nHif368Cg7kckOxznnEu4yDW0zWyFpHQzKwMekfRhAuNKac98tIbNu4qZ4H0VzrkGImqy2BOOaJon\n6fcEnd4NsupscWk597+Xx5BebRnWu32yw3HOuVoRtRnqmwQjmiYAuwnmT1yUqKBS2Ysf57Nhxz4m\nnNYv2aE451ytiVpIcHX4dC/w68SFk9pKy8r5x7SVHNutDaf065DscJxzrtbEKyRYMc+hSmGZjgbj\nXws2sHrLHu7/5mBSvGK6c87VqHh3FueFP78f/qwoLDge2JOQiFJUeblx75QV9M9syZlHZSY7HOec\nq1XVJouK5idJI81sZMxLt0uaDtyZyOBSyZuLN7K8YBd/vnwQaWl+V+Gca1gir8EtaVTFRrg4UYMZ\nDWVm/G3KCnq1b855x3VNdjjOOVfrog6dvR54WFKbcHs7wVKrDcLUZYV8sm4nv7/oONL9rsI51wBF\nHQ01BzheUmuCelI7EhtW6jAz7n13BV3bNOWCE7olOxznnEuKeKOhrjKzJyTdWmk/AGZ2TwJjSwmz\nVm0ld/U2fn3+0TTOiNpq55xz9Uu8O4uKfolWiQ4kVf3t3RV0aNmEy4Z0j3+wc87VU/FGQ90f/myQ\nE/HmrtnGBys2c8c5A2jaKD3Z4TjnXNLEa4b6S3Wvm9nNNRtOarl3ygqOaN6I8cN7JjsU55xLqnjN\nUHNqJYoUtHj9Tt5eUsCPzuhPyyaRi/M651y9FK8Z6tHaCiTV3Dt1BS2bZHDNiF7JDsU555Iu0p/M\nkjoCPwUGAk0r9pvZaQmKK6lWFu7itYUb+O7oPrRp3ijZ4TjnXNJFHQv6JLAEyCaoOvsZMDveSZLG\nSloqaYWk26t4/RpJhZLmhY9vx7zWQ9KbkpZIWiypV8RYD9vfp6ykSUYa14/Krq2PdM65lBa1Mb69\nmT0k6RYzmwZMkzStuhMkpQP3AmcC+cBsSZPMbHGlQ//PzCZU8RaPAf9tZm9JagmUR4z1sKzduoeX\n563jWyf1pEPLJrXxkc45l/KiJouS8OcGSV8D1gNZcc4ZCqwwszwASc8A44DKyeIrJA0EMszsLQAz\n2xUxzsN237SVpEvceErv2vpI55xLeVGboX4T1oX6MfAT4EHgR3HO6QasjdnOD/dVdpGkBZKel1Qx\n860/sF3Si5LmSvpDeKfyJZJulJQrKbewsDDipRzYpp37eC43n4sGZ9GlTbPDfj/nnKsvoiaLWWa2\nw8w+MbMxZjbYzCbFOaeqinuVF1J6FegVLqL0NlAx+ioDOJkgMQ0BegPXfOXNzCaaWY6Z5XTs2DHi\npRzYxPfyKDPjptF9Dvu9nHOuPomaLD4MO5uvl9Q24jn5BGt1V8giaL76nJltMbP94eYDwOCYc+ea\nWZ6ZlQIvAydG/NxDsmXXfp6atYZxx3elR/vmifwo55yrcyIlCzPrB/wcOBqYI+lfkq6Kc9psoJ+k\nbEmNgcuBL92NSOoSs3k+wYirinPbhkN2AU4jQl/H4Xh4+ir2lZbxvTF+V+Gcc5VFLqNqZh+Z2a0E\nHddb+aLJ6EDHlwITgMkESeBZM1sk6U5J54eH3SxpkaT5wM2ETU1mVkbQBPVOuA64CO48EmLH3hIe\n+3A15xzTmb6dGmzNROecO6Cok/JaAxcS3B30AV4iSBrVMrPXgNcq7ftFzPM7gDsOcO5bwHFR4jtc\nj334GUX7S/n+mL618XHOOVfnRB06O5+g3+BOM5uRwHhq3e79pTw8fRWnDejE0V3bxD/BOecaoKjJ\noreZVR7JVC/s2l/KSX3ac/0on1fhnHMHEnVZ1XqZKAAyWzfl7+MHxz/QOecaMF8n1DnnXFyeLJxz\nzsUVKVlI+r2k1pIaSXpH0uYI8yycc87VE1HvLM4ys53AeQSzq/sDtyUsKueccyklarKoWAHoXOBp\nM9uaoHicc86loKhDZ1+V9CmwF/heWIZjX+LCcs45l0qi1oa6HTgJyDGzEmA3wdoUzjnnGoCoHdyX\nAKVmVibp58ATQNeERuaccy5lKMp8O0kLzOw4SaOA3wJ/BH5mZsMSHWBUkgqB1cmO4xB0ADYnO4ha\n5tfcMPg11w09zSzugkBR+yzKwp9fA/5hZq9I+tWhRpYIUS42FUnKNbOcZMdRm/yaGwa/5vol6mio\ndZLuBy4FXpPU5CDOdc45V8dF/cK/lGBdirFmth1oh8+zcM65BiPqaKg9wErgbEkTgE5m9mZCI2s4\nJiY7gCTwa24Y/Jrrkagd3LcANwAvhrsuBCaa2V8TGJtzzrkUEXk0FHCSme0Ot1sAM8ysVlayc845\nl1xR+yzEFyOiCJ+r5sNpOCR1lzRF0pJwHfJbkh1TbZCULmmupH8lO5baIOkISc9L+jT8tz4p2TEl\nmqQfhf9NfyLpaUlNkx1TTZP0sKQCSZ/E7Gsn6S1Jy8OfbZMZY02LmiweAWZJ+lU4ZHYm8FDComoY\nSoEfm9lRwHDg+5IGJjmm2nALsCTZQdSiPwNvmNkA4Hjq+bVL6gbcTFDt4RggHbg8uVElxD+BsZX2\n3Q68Y2b9gHfC7Xojagf3PcC1wFZgG3Ctmf0pkYHVd2a2wcw+Dp8XEXyJdEtuVIklKYtgrs6DyY6l\nNkhqDZxC+IeVmRWHownruwygmaQMoDmwPsnx1Dgze4/g+zDWOODR8PmjwAW1GlSCxZ2UJykNWBD+\nlfBx4kNqeCT1Ak4AZiU3koT7E/D/gFbJDqSW9AYKgUckHQ/MAW6p6Purj8xsnaQ/AmsICo++2YBG\nTmaa2QYI/hiU1CnZAdWkuHcWZlYOzJfUoxbiaXAktQReAH4YrhlSL0k6DygwsznJjqUWZQAnElQ9\nOIGgAGe9apqoLGynHwdkE9SPa+ELpdUPUfssugCLwlXyJlU8EhlYQyCpEUGieNLMXox3fB03Ejhf\n0mfAM8Bpkp5IbkgJlw/km1nFHePzBMmjPjsDWGVmhWGF6heBEUmOqbZsktQFIPxZkOR4alTU2lC/\nTmgUDZAkEbRlLwn7hOo1M7sDuANA0qnAT8ysXv/FaWYbJa2VdKSZLQVOBxYnO64EWwMMl9ScoBnq\ndCA3uSHVmknA1cBd4c9XkhtOzao2WUjqS9AON63S/lOAdYkMrAEYCXwTWChpXrjvZ2b2WhJjcjXv\nB8CTkhoDeQQDReotM5sl6XmC/s1SYC71cFazpKeBU4EOkvKBXxIkiWclXU+QNC9JXoQ1r9pJeeFY\n+J+Z2YJK+3OAX5rZ1xMcn3POuRQQr8+iV+VEAWBmuUCvhETknHMu5cRLFtXNvGxWk4E455xLXfGS\nxWxJN1TeGbbJNaQhkM4516DF67PIBF4CivkiOeQAjYELzWxjwiN0zjmXdFGrzo4Bjgk3F5nZuwmN\nyjnnXEqJWhtqipn9NXx4onBVkmSS7o7Z/klNrdUu6Z+SLq6J94rzOZeE1WGnVPHaH8Jqqn84hPcd\nJOncmokyMSTtOsTzLjiUIpiH+nkuOXwdbVeT9gPfkNQh2YHEkpR+EIdfD3zPzMZU8dp3gBPN7FCW\nFB4EHFSyUKAu/D96AdAQKiY3aHXhP0RXd5QSTMD6UeUXKt8ZVPxVKelUSdMkPStpmaS7JI2X9JGk\nhZL6xLzNGZLeD487Lzw/PfyLf7akBZK+E/O+UyQ9BSysIp4rwvf/RNLvwn2/AEYB91W+ewjL27Qg\nKNV/maSOkl4IP3e2pJHhcUMlfahgzY4PJR0ZTsi7E7hM0rzw/F9J+knM+38iqVf4WCLp7wQT27pL\nOkvSDEkfS3ourCdG+LtaHF73H6u4xtHh580L42kV7r8t5vdVZXWGAx0j6VvhvvmSHpc0Ajgf+EP4\nOX3CxxuS5oT/XgPCc7PD65gt6b+q+lyXwszMH/6okQewC2gNfAa0AX4C/Cp87Z/AxbHHhj9PBbYT\n1B9rQlAZ4Nfha7cAf4o5/w2CP3D6EdRdagrcCPw8PKYJQWmJ7PB9dwPZVcTZlWCGbUeCKgbvAheE\nr00lWIuhyuuLef4UMCp83oOgbAvh9WeEz88AXgifXwP8Leb8XxGUPKnY/oRg7lIvoBwYHu7vALwH\ntAi3fwr8AmgHLOWLfscjqoj3VWBk+LxleK1nESR0hb/LfwGnVPo3qfIY4OjwMzuEx7U7wL/tO0C/\n8Pkw4N3w+STgW+Hz78f+Pv2R+o+otaGci8TMdkp6jGABnL0RT5ttYWlnSSuBipLWC4HY5qBnLaiC\nvFxSHjCA4IvtuJi7ljYEyaQY+MjMVlXxeUOAqWZWGH7mkwRfhi9HjBeCRDBQ+nzByNbhX+5tgEcl\n9QMMaHQQ71lhtZnNDJ8PJ2jimR5+VmNgBrAT2Ac8KOnfBF/olU0H7gmv70Uzy5d0FsHvbG54TEuC\n39d7Mecd6JjjgefNbDOAmVVez6GiivII4LmY302T8OdI4KLw+ePA7+L+JlzK8GThEuFPBE0oj8Ts\nKyVs9lTwLdI45rX9Mc/LY7bL+fJ/o5WH7hnBX78/MLPJsS8oKFZ4oHUjamJJ4DSCdem/lBAl/RWY\nYmYXKlinZOoBzv/89xGKnQAbG7eAt8zsispvIGkoQaG+y4EJwGmxr5vZXWEiOReYKemM8P1+a2b3\nV3NtVR4j6Wa++m9QWRqw3cwGHeD1+MMvXUryPgtX48K/OJ8l6Cyu8BkwOHw+jkP7i/sSSWlhP0Zv\ngiaRycBNCsq9I6m/pBZx3mcWMFpSh7Dz+wpgWpxzKnuT4Aua8HMrvhzb8EWRzWtiji/iy4s+fUZY\nrlzSiQRNZ1WZCYxUUNQTSc3Da2wJtLGg8OQPCTrQv0RSHzNbaGa/I2ieG0Dw+7oupt+jm766SM+B\njnkHuFRS+3B/u8rXZsGaLKskXRIeIwULP0Fwp1OxxOr4A1yvS1GeLFyi3E3Q3l7hAYIv6I8I2rEP\nZbW4pQRf6q8D3zWzfQRLtC4GPpb0CXA/ce6YwyavO4ApwHzgYzM72HLSNwM5YWfvYuC74f7fA7+V\nNJ1g/ekKUwiareZJuoxgHZN2CioO3wQsO0CshQRJ52lJCwiSxwCCL+d/hfumUcWgAuCHYcf5fIIm\nwdctWLXuKWCGpIUEa2x8aeXCAx1jZouA/wamhe9ZUVr/GeC2sBO9D0EiuD48ZhHBHwcQ9EF9X9Js\ngqTq6pBIk/Kcc841bH5n4ZxzLi5PFs455+LyZOGccy4uTxbOOefi8mThnHMuLk8Wzjnn4vJk4Zxz\nLi5PFs455+L6/2pZKSwETXp1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10be76358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.feature_selection import RFECV\n",
    "from sklearn.datasets import make_classification\n",
    "\n",
    "X = data.drop('SeriousDlqin2yrs', axis=1)\n",
    "# Create the RFE object and compute a cross-validated score.\n",
    "svc = LogisticRegression(C=1e9)\n",
    "# The \"accuracy\" scoring is proportional to the number of correct\n",
    "# classifications\n",
    "rfecv = RFECV(estimator=svc, step=1, cv=StratifiedKFold(2),\n",
    "              scoring='roc_auc')\n",
    "rfecv.fit(X, y)\n",
    "\n",
    "print(\"Optimal number of features : %d\" % rfecv.n_features_)\n",
    "\n",
    "# Plot number of features VS. cross-validation scores\n",
    "plt.figure()\n",
    "plt.xlabel(\"Number of features selected\")\n",
    "plt.ylabel(\"Cross validation score (nb of correct classifications)\")\n",
    "plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}