xgboost_caip.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TWrdW148R9kP"
   },
   "source": [
    "# Cloud AI Platform + What-if Tool: Playground XGBoost Example\n",
    "\n",
    "This notebook shows how to use the [What-if Tool](https://pair-code.github.io/what-if-tool/) on a deployed [Cloud AI Platform](https://cloud.google.com/ai-platform/) model. *You don't need your own cloud project* to run this notebook. \n",
    "\n",
    "For instructions on creating a Cloud project, see the documentation [here](https://cloud.google.com/resource-manager/docs/creating-managing-projects)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "python_version = sys.version_info[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PzmV_odNPBTb"
   },
   "outputs": [],
   "source": [
    "# If you're running on Colab, you'll need to install the What-if Tool package and authenticate on the TF instance\n",
    "def pip_install(module):\n",
    "    if python_version == '2':\n",
    "        !pip install {module} --quiet\n",
    "    else:\n",
    "        !pip3 install {module} --quiet\n",
    "\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "\n",
    "if IN_COLAB:\n",
    "    pip_install('witwidget')\n",
    "\n",
    "    from google.colab import auth\n",
    "    auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CosDxuLy7M4Q"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import witwidget\n",
    "\n",
    "from witwidget.notebook.visualization import WitWidget, WitConfigBuilder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bFIxtguO1In_"
   },
   "source": [
    "## Loading the test dataset\n",
    "\n",
    "The model we'll be exploring here is a binary classification model built with XGBoost and trained on a [mortgage dataset](https://www.ffiec.gov/hmda/hmdaflat.htm). It predicts whether or not a mortgage application will be approved. In this section we'll:\n",
    "\n",
    "* Download some test data from Cloud Storage and load it into a numpy array + Pandas DataFrame\n",
    "* Preview the features for our model in Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9BngZjdsO6Mr"
   },
   "outputs": [],
   "source": [
    "# Download our Pandas dataframe and our test features and labels\n",
    "!gsutil cp gs://mortgage_dataset_files/data.pkl .\n",
    "!gsutil cp gs://mortgage_dataset_files/x_test.npy .\n",
    "!gsutil cp gs://mortgage_dataset_files/y_test.npy ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GkHavVlmGYlk"
   },
   "outputs": [],
   "source": [
    "# Preview the features from our model as a pandas DataFrame\n",
    "features = pd.read_pickle('data.pkl')\n",
    "features.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "57KQ_XX2FEdl"
   },
   "outputs": [],
   "source": [
    "# Load the test features and labels into numpy ararys\n",
    "x_test = np.load('x_test.npy')\n",
    "y_test = np.load('y_test.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hoFIrCQfFgvm"
   },
   "outputs": [],
   "source": [
    "# Combine the features and labels into one array for the What-if Tool\n",
    "test_examples = np.hstack((x_test,y_test.reshape(-1,1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-8xNn8EhgUi7"
   },
   "source": [
    "## Using the What-if Tool to interpret our model\n",
    "With our test examples ready, we can now connect our model to the What-if Tool using the `WitWidget`. To use the What-if Tool with Cloud AI Platform, we need to send it:\n",
    "* A Python list of our test features + ground truth labels\n",
    "* Optionally, the names of our columns\n",
    "* Our Cloud project, model, and version name (we've created a public one for you to play around with)\n",
    "\n",
    "See the next cell for some exploration ideas in the What-if Tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dqAbAmxkgW4p"
   },
   "outputs": [],
   "source": [
    "# Create a What-if Tool visualization, it may take a minute to load\n",
    "# See the cell below this for exploration ideas\n",
    "\n",
    "# This prediction adjustment function is needed as this xgboost model's\n",
    "# prediction returns just a score for the positive class of the binary\n",
    "# classification, whereas the What-If Tool expects a list of scores for each\n",
    "# class (in this case, both the negative class and the positive class).\n",
    "\n",
    "def adjust_prediction(pred):\n",
    "  return [1 - pred, pred]\n",
    "\n",
    "config_builder = (WitConfigBuilder(test_examples.tolist(), features.columns.tolist() + ['mortgage_status'])\n",
    "  .set_ai_platform_model('wit-caip-demos', 'xgb_mortgage', 'v1', adjust_prediction=adjust_prediction)\n",
    "  .set_target_feature('mortgage_status')\n",
    "  .set_label_vocab(['denied', 'approved']))\n",
    "WitWidget(config_builder, height=800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_B2BskDk55rk"
   },
   "source": [
    "## What-if Tool exploration ideas\n",
    "\n",
    "* **Individual data points**: the default graph shows all data points from the test set, colored by their ground truth label (approved or denied)\n",
    "  * Try selecting data points close to the middle and tweaking some of their feature values. Then run inference again to see if the model prediction changes\n",
    "  * Select a data point and then move the \"Show nearest counterfactual datapoint\" slider to the right. This will highlight a data point with feature values closest to your original one, but with a different prediction\n",
    "  \n",
    "* **Binning data**: create separate graphs for individual features\n",
    "  * From the \"Binning - X axis\" dropdown, try selecting one of the agency codes, for example \"Department of Housing and Urban Development (HUD)\". This will create 2 separate graphs, one for loan applications from the HUD (graph labeled 1), and one for all other agencies (graph labeled 0). This shows us that loans from this agency are more likely to be denied\n",
    "\n",
    "* **Exploring overall performance**: Click on the \"Performance & Fairness\" tab to view overall performance statistics on the model's results on the provided dataset, including confusion matrices, PR curves, and ROC curves.\n",
    "   * Experiment with the threshold slider, raising and lowering the positive classification score the model needs to return before it decides to predict \"approved\" for the loan, and see how it changes accuracy, false positives, and false negatives.\n",
    "   * On the left side \"Slice by\" menu, select \"loan_purpose_Home purchase\". You'll now see performance on the two subsets of your data: the \"0\" slice shows when the loan is not for a home purchase, and the \"1\" slice is for when the loan is for a home purchase. Notice that the model's false positive rate is much higher on loans for home purchases. If you expand the rows to look at the confusion matrices, you can see that the model predicts \"approved\" more often for home purchase loans.\n",
    "   * You can use the optimization buttons on the left side to have the tool auto-select different positive classification thresholds for each slice in order to achieve different goals. If you select the \"Demographic parity\" button, then the two thresholds will be adjusted so that the model predicts \"approved\" for a similar percentage of applicants in both slices. What does this do to the accuracy, false positives and false negatives for each slice?\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "What-If Tool with XGBoost Cloud AI Platform Model",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}