LightAutoML demo (Whitebox).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AutoWoE (WhiteBox модель для бинарной классификации на табличных данных)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Скор карта"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![WB0](imgs/tutorial_whitebox_report_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Линейная модель"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![WB1](imgs/tutorial_whitebox_report_2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Дискретизация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![WB2](imgs/tutorial_whitebox_report_3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Отбор и одномерный анализ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![WB3](imgs/tutorial_whitebox_report_4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Whitebox pipeline:\n",
    "\n",
    "###  Общие параметры:\n",
    "\n",
    "    - n_jobs\n",
    "    - debug\n",
    "\n",
    "### 0) Простая разметка типов и выброс мусора\n",
    "####    0.0) Удаление мусорных фичей\n",
    "    \n",
    "        Medium:\n",
    "        - th_nan \n",
    "        - th_const \n",
    "        \n",
    "####    0.1) Разметка типов (авто или пользовательская)\n",
    "        \n",
    "        Critical:\n",
    "        - features_type (dict) {'age': 'real', 'education': 'cat', 'birth_date': (None, (\"d\", \"wd\"), ...}\n",
    "        \n",
    "####    0.2) Кодирование категорий и дат\n",
    "        \n",
    "        Critical:\n",
    "        - features_type (for datetimes)\n",
    "        \n",
    "        Optional:\n",
    "        - cat_alpha (int) - greater means more conservative encoding\n",
    "    \n",
    "        \n",
    "### 1) Первая стадия отбора (отбор на основе важности BlackBox модели)\n",
    "\n",
    "    Critical:\n",
    "    - select_type (None or int)\n",
    "    - imp_type (if type(select_type) is int 'perm_imt'/'feature_imp') \n",
    "    \n",
    "    Optional:\n",
    "    - imt_th (float) - threshold for select_type is None\n",
    "    \n",
    "### 2) Биннинг (дискретизация):\n",
    "    \n",
    "    Critical:\n",
    "    - monotonic / features_monotone_constraints \n",
    "    - max_bin_count / max_bin_count\n",
    "    - min_bin_size\n",
    "    \n",
    "    - cat_merge_to\n",
    "    - nan_merge_to\n",
    "    \n",
    "    Medium:\n",
    "    - force_single_split\n",
    "    \n",
    "    Optional:\n",
    "    - min_bin_mults\n",
    "    - min_gains_to_split\n",
    "\n",
    "### 3) WoE оценки WoE = LN( ((% 0 in bin) / (% 0 in sample)) / ((% 1 in bin) / (% 1 in sample)) ):\n",
    "    \n",
    "    Critical:\n",
    "    - oof_woe\n",
    "    \n",
    "    Optional:\n",
    "    - woe_diff_th\n",
    "    - n_folds (if oof_woe)\n",
    "\n",
    "### 4) Вторая стадия отбора:\n",
    "\n",
    "####    4.0) Одномерная зависимость с таргетом\n",
    "    \n",
    "    Critical:\n",
    "    - auc_th\n",
    "    \n",
    "####    4.1) VIF \n",
    "    \n",
    "    Critical:\n",
    "    - vif_th\n",
    "    \n",
    "####    4.2) Частные корреляции\n",
    "    \n",
    "    Critical:\n",
    "    - pearson_th\n",
    "    \n",
    "### 5) Третья стадия отбора (на основе модели)\n",
    "    \n",
    "    Optional:\n",
    "    - n_folds\n",
    "    - l1_base_step\n",
    "    - l1_exp_step\n",
    "    \n",
    "    Do not touch:\n",
    "    - population_size\n",
    "    - feature_groups_count\n",
    "\n",
    "\n",
    "### 6) Обучение финальной модели:\n",
    "\n",
    "    Critical:\n",
    "    - regularized_refit\n",
    "    - p_val (if not regularized_refit)\n",
    "    - validation (if not regularized_refit)\n",
    "    \n",
    "    Optional:\n",
    "    - interpreted_model\n",
    "    - l1_base_step (if regularized_refit)\n",
    "    - l1_exp_step (if regularized_refit)\n",
    "    \n",
    "### 7) Создание отчета\n",
    "\n",
    "    - report_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Импорты"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import joblib\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "from autowoe import AutoWoE, ReportDeco"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Чтение данных и train/test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('example_data/jobs_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enrollee_id</th>\n",
       "      <th>city</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>gender</th>\n",
       "      <th>relevent_experience</th>\n",
       "      <th>enrolled_university</th>\n",
       "      <th>education_level</th>\n",
       "      <th>major_discipline</th>\n",
       "      <th>experience</th>\n",
       "      <th>company_size</th>\n",
       "      <th>company_type</th>\n",
       "      <th>last_new_job</th>\n",
       "      <th>training_hours</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8949</td>\n",
       "      <td>city_103</td>\n",
       "      <td>0.920</td>\n",
       "      <td>Male</td>\n",
       "      <td>Has relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>STEM</td>\n",
       "      <td>21.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>36</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>29725</td>\n",
       "      <td>city_40</td>\n",
       "      <td>0.776</td>\n",
       "      <td>Male</td>\n",
       "      <td>No relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>STEM</td>\n",
       "      <td>15.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Pvt Ltd</td>\n",
       "      <td>5.0</td>\n",
       "      <td>47</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11561</td>\n",
       "      <td>city_21</td>\n",
       "      <td>0.624</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No relevent experience</td>\n",
       "      <td>Full time course</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>STEM</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>83</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>33241</td>\n",
       "      <td>city_115</td>\n",
       "      <td>0.789</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No relevent experience</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>Business Degree</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pvt Ltd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>52</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>666</td>\n",
       "      <td>city_162</td>\n",
       "      <td>0.767</td>\n",
       "      <td>Male</td>\n",
       "      <td>Has relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Masters</td>\n",
       "      <td>STEM</td>\n",
       "      <td>21.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Funded Startup</td>\n",
       "      <td>4.0</td>\n",
       "      <td>8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19153</th>\n",
       "      <td>7386</td>\n",
       "      <td>city_173</td>\n",
       "      <td>0.878</td>\n",
       "      <td>Male</td>\n",
       "      <td>No relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>Humanities</td>\n",
       "      <td>14.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>42</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19154</th>\n",
       "      <td>31398</td>\n",
       "      <td>city_103</td>\n",
       "      <td>0.920</td>\n",
       "      <td>Male</td>\n",
       "      <td>Has relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>STEM</td>\n",
       "      <td>14.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>52</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19155</th>\n",
       "      <td>24576</td>\n",
       "      <td>city_103</td>\n",
       "      <td>0.920</td>\n",
       "      <td>Male</td>\n",
       "      <td>Has relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Graduate</td>\n",
       "      <td>STEM</td>\n",
       "      <td>21.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>Pvt Ltd</td>\n",
       "      <td>4.0</td>\n",
       "      <td>44</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19156</th>\n",
       "      <td>5756</td>\n",
       "      <td>city_65</td>\n",
       "      <td>0.802</td>\n",
       "      <td>Male</td>\n",
       "      <td>Has relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>High School</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>999.0</td>\n",
       "      <td>Pvt Ltd</td>\n",
       "      <td>2.0</td>\n",
       "      <td>97</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19157</th>\n",
       "      <td>23834</td>\n",
       "      <td>city_67</td>\n",
       "      <td>0.855</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No relevent experience</td>\n",
       "      <td>no_enrollment</td>\n",
       "      <td>Primary School</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>127</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19158 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       enrollee_id      city  city_development_index gender  \\\n",
       "0             8949  city_103                   0.920   Male   \n",
       "1            29725   city_40                   0.776   Male   \n",
       "2            11561   city_21                   0.624    NaN   \n",
       "3            33241  city_115                   0.789    NaN   \n",
       "4              666  city_162                   0.767   Male   \n",
       "...            ...       ...                     ...    ...   \n",
       "19153         7386  city_173                   0.878   Male   \n",
       "19154        31398  city_103                   0.920   Male   \n",
       "19155        24576  city_103                   0.920   Male   \n",
       "19156         5756   city_65                   0.802   Male   \n",
       "19157        23834   city_67                   0.855    NaN   \n",
       "\n",
       "           relevent_experience enrolled_university education_level  \\\n",
       "0      Has relevent experience       no_enrollment        Graduate   \n",
       "1       No relevent experience       no_enrollment        Graduate   \n",
       "2       No relevent experience    Full time course        Graduate   \n",
       "3       No relevent experience                 NaN        Graduate   \n",
       "4      Has relevent experience       no_enrollment         Masters   \n",
       "...                        ...                 ...             ...   \n",
       "19153   No relevent experience       no_enrollment        Graduate   \n",
       "19154  Has relevent experience       no_enrollment        Graduate   \n",
       "19155  Has relevent experience       no_enrollment        Graduate   \n",
       "19156  Has relevent experience       no_enrollment     High School   \n",
       "19157   No relevent experience       no_enrollment  Primary School   \n",
       "\n",
       "      major_discipline  experience  company_size    company_type  \\\n",
       "0                 STEM        21.0           NaN             NaN   \n",
       "1                 STEM        15.0          99.0         Pvt Ltd   \n",
       "2                 STEM         5.0           NaN             NaN   \n",
       "3      Business Degree         0.0           NaN         Pvt Ltd   \n",
       "4                 STEM        21.0          99.0  Funded Startup   \n",
       "...                ...         ...           ...             ...   \n",
       "19153       Humanities        14.0           NaN             NaN   \n",
       "19154             STEM        14.0           NaN             NaN   \n",
       "19155             STEM        21.0          99.0         Pvt Ltd   \n",
       "19156              NaN         0.0         999.0         Pvt Ltd   \n",
       "19157              NaN         2.0           NaN             NaN   \n",
       "\n",
       "       last_new_job  training_hours  target  \n",
       "0               1.0              36     1.0  \n",
       "1               5.0              47     0.0  \n",
       "2               0.0              83     0.0  \n",
       "3               0.0              52     1.0  \n",
       "4               4.0               8     0.0  \n",
       "...             ...             ...     ...  \n",
       "19153           1.0              42     1.0  \n",
       "19154           4.0              52     1.0  \n",
       "19155           4.0              44     0.0  \n",
       "19156           2.0              97     0.0  \n",
       "19157           1.0             127     0.0  \n",
       "\n",
       "[19158 rows x 14 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = train_test_split(data.drop('enrollee_id', axis=1), test_size=0.2, stratify=data['target'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AutoWoe: настройки по умолчанию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_woe_0 = AutoWoE(interpreted_model=True,\n",
    "                     monotonic=False,\n",
    "                     max_bin_count=5,\n",
    "                     select_type=None,\n",
    "                     pearson_th=0.9,\n",
    "                     auc_th=.505,\n",
    "                     vif_th=10.,\n",
    "                     imp_th=0,\n",
    "                     th_const=32,\n",
    "                     force_single_split=True,\n",
    "                     th_nan=0.01,\n",
    "                     th_cat=0.005,\n",
    "                     auc_tol=1e-4,\n",
    "                     cat_alpha=100,\n",
    "                     cat_merge_to=\"to_woe_0\",\n",
    "                     nan_merge_to=\"to_woe_0\",\n",
    "                     imp_type=\"feature_imp\",\n",
    "                     regularized_refit=False,\n",
    "                     p_val=0.05,\n",
    "                     verbose=2\n",
    "        )\n",
    "\n",
    "auto_woe_0 = ReportDeco(auto_woe_0, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "city processing...\n",
      "city_development_index processing...\n",
      "gender processing...\n",
      "relevent_experience processing...\n",
      "enrolled_university processing...\n",
      "education_level processing...\n",
      "experience processing...\n",
      "company_size processing...\n",
      "company_type processing...\n",
      "last_new_job processing...\n",
      "training_hours processing...\n",
      "dict_keys(['city', 'city_development_index', 'gender', 'relevent_experience', 'enrolled_university', 'education_level', 'experience', 'company_size', 'company_type', 'last_new_job', 'training_hours']) to selector !!!!!\n",
      "Feature selection...\n",
      "city_development_index   -0.974107\n",
      "company_size             -0.795953\n",
      "company_type             -0.400146\n",
      "experience               -0.184238\n",
      "enrolled_university      -0.251287\n",
      "education_level          -1.188926\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "auto_woe_0.fit(train,\n",
    "               target_name=\"target\",\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.06265852, 0.56483877, 0.04151965, ..., 0.15191705, 0.08528486,\n",
       "       0.0409943 ])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_prediction = auto_woe_0.predict_proba(test)\n",
    "test_prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8034365349304012"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(test['target'].values, test_prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n"
     ]
    }
   ],
   "source": [
    "report_params = {\"output_path\": \"HR_REPORT_1\", # папка, куда сгенерится отчет и сложатся нужные файлы\n",
    "                 \"report_name\": \"WHITEBOX REPORT\",\n",
    "                 \"report_version_id\": 1,\n",
    "                 \"city\": \"Moscow\",\n",
    "                 \"model_aim\": \"Predict if candidate will work for the company\",\n",
    "                 \"model_name\": \"HR model\",\n",
    "                 \"zakazchik\": \"Kaggle\",\n",
    "                 \"high_level_department\": \"Ai Lab\",\n",
    "                 \"ds_name\": \"Btbpanda\",\n",
    "                 \"target_descr\": \"Candidate will work for the company\",\n",
    "                 \"non_target_descr\": \"Candidate will work for the company\"}\n",
    "\n",
    "auto_woe_0.generate_report(report_params, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AutoWoE - более консервативная модель"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "auto_woe_1 = AutoWoE(interpreted_model=True,\n",
    "                     monotonic=True,\n",
    "                     max_bin_count=4,\n",
    "                     select_type=None,\n",
    "                     pearson_th=0.9,\n",
    "                     auc_th=.505,\n",
    "                     vif_th=10.,\n",
    "                     imp_th=0,\n",
    "                     th_const=32,\n",
    "                     force_single_split=True,\n",
    "                     th_nan=0.01,\n",
    "                     th_cat=0.005,\n",
    "                     auc_tol=1e-4,\n",
    "                     cat_alpha=100,\n",
    "                     cat_merge_to=\"to_woe_0\",\n",
    "                     nan_merge_to=\"to_woe_0\",\n",
    "                     imp_type=\"feature_imp\",\n",
    "                     regularized_refit=False,\n",
    "                     p_val=0.05,\n",
    "                     verbose=2\n",
    "        )\n",
    "\n",
    "auto_woe_1 = ReportDeco(auto_woe_1, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "city processing...city_development_index processing...\n",
      "\n",
      "gender processing...\n",
      "relevent_experience processing...\n",
      "enrolled_university processing...education_level processing...\n",
      "\n",
      "experience processing...company_type processing...company_size processing...\n",
      "\n",
      "\n",
      "last_new_job processing...\n",
      "training_hours processing...\n",
      "dict_keys(['city', 'city_development_index', 'gender', 'relevent_experience', 'enrolled_university', 'education_level', 'experience', 'company_size', 'company_type', 'last_new_job', 'training_hours']) to selector !!!!!\n",
      "Feature selection...\n",
      "city                     -0.516274\n",
      "city_development_index   -0.512608\n",
      "company_size             -0.814922\n",
      "company_type             -0.397978\n",
      "experience               -0.175231\n",
      "enrolled_university      -0.219507\n",
      "education_level          -1.239627\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "auto_woe_1.fit(train,\n",
    "               target_name=\"target\",\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.06460692, 0.57321671, 0.0497262 , ..., 0.13746553, 0.07190761,\n",
       "       0.04153373])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_prediction = auto_woe_1.predict_proba(test)\n",
    "test_prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8019815944109903"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(test['target'].values, test_prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n",
      "No handles with labels found to put in legend.\n"
     ]
    }
   ],
   "source": [
    "report_params = {\"output_path\": \"HR_REPORT_2\", # папка, куда сгенерится отчет и сложатся нужные файлы\n",
    "                 \"report_name\": \"WHITEBOX REPORT\",\n",
    "                 \"report_version_id\": 2,\n",
    "                 \"city\": \"Moscow\",\n",
    "                 \"model_aim\": \"Predict if candidate will work for the company\",\n",
    "                 \"model_name\": \"HR model\",\n",
    "                 \"zakazchik\": \"Kaggle\",\n",
    "                 \"high_level_department\": \"Ai Lab\",\n",
    "                 \"ds_name\": \"Btbpanda\",\n",
    "                 \"target_descr\": \"Candidate will work for the company\",\n",
    "                 \"non_target_descr\": \"Candidate will work for the company\"}\n",
    "\n",
    "auto_woe_1.generate_report(report_params, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WhiteBox preset - использование по аналогии с TabularAutoML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightautoml.automl.presets.whitebox_presets import WhiteBoxPreset\n",
    "from lightautoml import Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = Task('binary')\n",
    "automl = WhiteBoxPreset(task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Validation data is not set. Train will be used as valid in report and valid prediction\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start automl preset with listed constraints:\n",
      "- time: 3600 seconds\n",
      "- cpus: 4 cores\n",
      "- memory: 16 gb\n",
      "\n",
      "Train data shape: (15326, 13)\n",
      "Feats was rejected during automatic roles guess: []\n",
      "\n",
      "\n",
      "Layer 1 ...\n",
      "Train process start. Time left 3595.0072581768036 secs\n",
      "Start fitting Lvl_0_Pipe_0_Mod_0_WhiteBox ...\n",
      "\n",
      "===== Start working with fold 0 for Lvl_0_Pipe_0_Mod_0_WhiteBox =====\n",
      "\n",
      " features [] contain too many nans or identical values\n",
      " features [] have low importance\n",
      "city processing...\n",
      "city_development_index processing...company_type processing...education_level processing...\n",
      "\n",
      "\n",
      "enrolled_university processing...\n",
      "gender processing...\n",
      "major_discipline processing...\n",
      "relevent_experience processing...\n",
      "company_size processing...\n",
      "experience processing...\n",
      "last_new_job processing...\n",
      "training_hours processing...\n",
      "dict_keys(['city', 'city_development_index', 'company_type', 'education_level', 'enrolled_university', 'gender', 'major_discipline', 'relevent_experience', 'company_size', 'experience', 'last_new_job', 'training_hours']) to selector !!!!!\n",
      "Feature selection...\n",
      "Feature training_hours removed due to low AUC value 0.5031265374717342\n",
      "Feature city_development_index removed due to high VIF value = 40.56438648184099\n",
      "C parameter range in [0.0002603488674824265:260.3488674824265], 20 values\n",
      "Result(score=0.7856775296767177, reg_alpha=0.020431136952654548, is_neg=True, min_weights=city                  -0.980620\n",
      "company_size          -0.800535\n",
      "company_type          -0.340185\n",
      "experience            -0.198176\n",
      "enrolled_university   -0.101047\n",
      "relevent_experience    0.000000\n",
      "education_level       -0.624324\n",
      "last_new_job           0.000000\n",
      "gender                 0.000000\n",
      "major_discipline      -0.317699\n",
      "dtype: float64)\n",
      "Iter 0 of final refit starts with 7 features\n",
      "Validation data checks\n",
      "city                  -0.956550\n",
      "company_size          -0.866063\n",
      "company_type          -0.402941\n",
      "experience            -0.329493\n",
      "enrolled_university   -0.230776\n",
      "education_level       -0.641994\n",
      "major_discipline      -1.596907\n",
      "dtype: float64\n",
      "Lvl_0_Pipe_0_Mod_0_WhiteBox fitting and predicting completed\n",
      "Time left 3587.2280378341675\n",
      "\n",
      "Automl preset training completed in 12.77 seconds.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "train_pred = automl.fit_predict(train.reset_index(drop=True), roles={'target': 'target'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_prediction = automl.predict(test).data[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7966826628232216"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(test['target'].values, test_prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Сериализация модели\n",
    "\n",
    "Важно: auto_woe_1 фактически является ReportDeco объектом (отчетом), не AutoWoE. Чтобы получить AutoWoE надо обратиться к атрибуту .model. \n",
    "\n",
    "ReportDeco не рекомендуется для использования на стадии инференса. Отчет требует целевой переменной в датасете для предсказания, так как считает метрики качества. Так же инференс из объекта-отчета намного дольше из-за собственно построения отчета."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "joblib.dump(auto_woe_1.model, 'model.pkl')\n",
    "model = joblib.load('model.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SQL запрос для инференса"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SELECT\n",
      "  1 / (1 + EXP(-(\n",
      "    -1.111\n",
      "    -0.516*WOE_TAB.city\n",
      "    -0.513*WOE_TAB.city_development_index\n",
      "    -0.815*WOE_TAB.company_size\n",
      "    -0.398*WOE_TAB.company_type\n",
      "    -0.175*WOE_TAB.experience\n",
      "    -0.22*WOE_TAB.enrolled_university\n",
      "    -1.24*WOE_TAB.education_level\n",
      "  ))) as PROB,\n",
      "  WOE_TAB.*\n",
      "FROM \n",
      "    (SELECT\n",
      "    CASE\n",
      "      WHEN (city IS NULL OR LOWER(CAST(city AS VARCHAR(50))) = 'nan') THEN 0\n",
      "      WHEN city IN ('city_100', 'city_102', 'city_103', 'city_116', 'city_149', 'city_159', 'city_160', 'city_45', 'city_46', 'city_64', 'city_71', 'city_73', 'city_83', 'city_99') THEN 0.213\n",
      "      WHEN city IN ('city_104', 'city_114', 'city_136', 'city_138', 'city_16', 'city_173', 'city_23', 'city_28', 'city_36', 'city_50', 'city_57', 'city_61', 'city_65', 'city_67', 'city_75', 'city_97') THEN 1.017\n",
      "      WHEN city IN ('city_11', 'city_21', 'city_74') THEN -1.455\n",
      "      ELSE -0.209\n",
      "    END AS city,\n",
      "    CASE\n",
      "      WHEN (city_development_index IS NULL OR city_development_index = 'NaN') THEN 0\n",
      "      WHEN city_development_index <= 0.6245 THEN -1.454\n",
      "      WHEN city_development_index <= 0.7915 THEN -0.121\n",
      "      WHEN city_development_index <= 0.9235 THEN 0.461\n",
      "      ELSE 1.101\n",
      "    END AS city_development_index,\n",
      "    CASE\n",
      "      WHEN (company_size IS NULL OR company_size = 'NaN') THEN -0.717\n",
      "      WHEN company_size <= 74.0 THEN 0.221\n",
      "      ELSE 0.467\n",
      "    END AS company_size,\n",
      "    CASE\n",
      "      WHEN (company_type IS NULL OR LOWER(CAST(company_type AS VARCHAR(50))) = 'nan') THEN -0.64\n",
      "      WHEN company_type IN ('Early Stage Startup', 'NGO', 'Other', 'Public Sector') THEN 0.164\n",
      "      WHEN company_type = 'Funded Startup' THEN 0.737\n",
      "      WHEN company_type = 'Pvt Ltd' THEN 0.398\n",
      "      ELSE 0\n",
      "    END AS company_type,\n",
      "    CASE\n",
      "      WHEN (experience IS NULL OR experience = 'NaN') THEN 0\n",
      "      WHEN experience <= 1.5 THEN -0.811\n",
      "      WHEN experience <= 7.5 THEN -0.319\n",
      "      WHEN experience <= 11.5 THEN 0.119\n",
      "      ELSE 0.533\n",
      "    END AS experience,\n",
      "    CASE\n",
      "      WHEN (enrolled_university IS NULL OR LOWER(CAST(enrolled_university AS VARCHAR(50))) = 'nan') THEN -0.327\n",
      "      WHEN enrolled_university = 'Full time course' THEN -0.614\n",
      "      WHEN enrolled_university = 'Part time course' THEN 0.026\n",
      "      WHEN enrolled_university = 'no_enrollment' THEN 0.208\n",
      "      ELSE 0\n",
      "    END AS enrolled_university,\n",
      "    CASE\n",
      "      WHEN (education_level IS NULL OR LOWER(CAST(education_level AS VARCHAR(50))) = 'nan') THEN 0.21\n",
      "      WHEN education_level = 'Graduate' THEN -0.166\n",
      "      WHEN education_level = 'High School' THEN 0.34\n",
      "      WHEN education_level = 'Masters' THEN 0.21\n",
      "      WHEN education_level IN ('Phd', 'Primary School') THEN 0.704\n",
      "      ELSE 0\n",
      "    END AS education_level\n",
      "  FROM global_temp.TABLE_1) as WOE_TAB\n"
     ]
    }
   ],
   "source": [
    "sql_query = model.get_sql_inference_query('global_temp.TABLE_1')\n",
    "print(sql_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Проверка SQL с использованием PySpark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder \\\n",
    "                    .master(\"local[2]\") \\\n",
    "                    .appName(\"spark-course\") \\\n",
    "                    .config(\"spark.driver.memory\", \"512m\") \\\n",
    "                    .getOrCreate()\n",
    "sc = spark.sparkContext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark_df = spark.read.csv(\"jobs_train.csv\", header=True)\n",
    "spark_df.createGlobalTempView(\"TABLE_1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = spark.sql(sql_query).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PROB</th>\n",
       "      <th>city</th>\n",
       "      <th>city_development_index</th>\n",
       "      <th>company_size</th>\n",
       "      <th>company_type</th>\n",
       "      <th>experience</th>\n",
       "      <th>enrolled_university</th>\n",
       "      <th>education_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.365512</td>\n",
       "      <td>0.213</td>\n",
       "      <td>0.461</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>-0.640</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.195716</td>\n",
       "      <td>-0.209</td>\n",
       "      <td>-0.121</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.398</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.835002</td>\n",
       "      <td>-1.455</td>\n",
       "      <td>-1.454</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>-0.640</td>\n",
       "      <td>-0.319</td>\n",
       "      <td>-0.614</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.476161</td>\n",
       "      <td>-0.209</td>\n",
       "      <td>-0.121</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>0.398</td>\n",
       "      <td>-0.811</td>\n",
       "      <td>-0.327</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.117694</td>\n",
       "      <td>-0.209</td>\n",
       "      <td>-0.121</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.737</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>0.210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19153</th>\n",
       "      <td>0.275602</td>\n",
       "      <td>1.017</td>\n",
       "      <td>0.461</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>-0.640</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19154</th>\n",
       "      <td>0.365512</td>\n",
       "      <td>0.213</td>\n",
       "      <td>0.461</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>-0.640</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19155</th>\n",
       "      <td>0.126794</td>\n",
       "      <td>0.213</td>\n",
       "      <td>0.461</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.398</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.208</td>\n",
       "      <td>-0.166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19156</th>\n",
       "      <td>0.060842</td>\n",
       "      <td>1.017</td>\n",
       "      <td>0.461</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.398</td>\n",
       "      <td>-0.811</td>\n",
       "      <td>0.208</td>\n",
       "      <td>0.340</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19157</th>\n",
       "      <td>0.130552</td>\n",
       "      <td>1.017</td>\n",
       "      <td>0.461</td>\n",
       "      <td>-0.717</td>\n",
       "      <td>-0.640</td>\n",
       "      <td>-0.319</td>\n",
       "      <td>0.208</td>\n",
       "      <td>0.704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19158 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           PROB    city city_development_index company_size company_type  \\\n",
       "0      0.365512   0.213                  0.461       -0.717       -0.640   \n",
       "1      0.195716  -0.209                 -0.121        0.467        0.398   \n",
       "2      0.835002  -1.455                 -1.454       -0.717       -0.640   \n",
       "3      0.476161  -0.209                 -0.121       -0.717        0.398   \n",
       "4      0.117694  -0.209                 -0.121        0.467        0.737   \n",
       "...         ...     ...                    ...          ...          ...   \n",
       "19153  0.275602   1.017                  0.461       -0.717       -0.640   \n",
       "19154  0.365512   0.213                  0.461       -0.717       -0.640   \n",
       "19155  0.126794   0.213                  0.461        0.467        0.398   \n",
       "19156  0.060842   1.017                  0.461        0.467        0.398   \n",
       "19157  0.130552   1.017                  0.461       -0.717       -0.640   \n",
       "\n",
       "      experience enrolled_university education_level  \n",
       "0          0.533               0.208          -0.166  \n",
       "1          0.533               0.208          -0.166  \n",
       "2         -0.319              -0.614          -0.166  \n",
       "3         -0.811              -0.327          -0.166  \n",
       "4          0.533               0.208           0.210  \n",
       "...          ...                 ...             ...  \n",
       "19153      0.533               0.208          -0.166  \n",
       "19154      0.533               0.208          -0.166  \n",
       "19155      0.533               0.208          -0.166  \n",
       "19156     -0.811               0.208           0.340  \n",
       "19157     -0.319               0.208           0.704  \n",
       "\n",
       "[19158 rows x 8 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.36557352, 0.19577798, 0.83497665, ..., 0.12678668, 0.06083813,\n",
       "       0.13061427])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "full_prediction = model.predict_proba(data)\n",
    "full_prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0002878641803194526"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(res['PROB'] - full_prediction).abs().max()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}