WIT_Model_Comparison.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "WIT Model Comparison",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UiNxsd4_q9wq",
        "colab_type": "text"
      },
      "source": [
        "### What-If Tool - model comparison\n",
        "\n",
        "Copyright 2019 Google LLC.\n",
        "SPDX-License-Identifier: Apache-2.0\n",
        "\n",
        "This notebook shows use of the [What-If Tool](https://pair-code.github.io/what-if-tool) to compare perfomance of two models on the same dataset.\n",
        "\n",
        "This notebook trains a linear classifier and a DNN on the [UCI census problem](https://archive.ics.uci.edu/ml/datasets/census+income) (predicting whether a person earns more than $50K from their census information).\n",
        "\n",
        "It then visualizes the results of the trained classifiers on test data using the What-If Tool.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qqB2tjOMETmr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Install the What-If Tool widget if running in colab {display-mode: \"form\"}\n",
        "\n",
        "try:\n",
        "  import google.colab\n",
        "  !pip install --upgrade witwidget\n",
        "except:\n",
        "  pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jlwjF-Nnmoww",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Define helper functions {display-mode: \"form\"}\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "import functools\n",
        "\n",
        "# Creates a tf feature spec from the dataframe and columns specified.\n",
        "def create_feature_spec(df, columns=None):\n",
        "    feature_spec = {}\n",
        "    if columns == None:\n",
        "        columns = df.columns.values.tolist()\n",
        "    for f in columns:\n",
        "        if df[f].dtype is np.dtype(np.int64):\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.int64)\n",
        "        elif df[f].dtype is np.dtype(np.float64):\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.float32)\n",
        "        else:\n",
        "            feature_spec[f] = tf.io.FixedLenFeature(shape=(), dtype=tf.string)\n",
        "    return feature_spec\n",
        "\n",
        "# Creates simple numeric and categorical feature columns from a feature spec and a\n",
        "# list of columns from that spec to use.\n",
        "#\n",
        "# NOTE: Models might perform better with some feature engineering such as bucketed\n",
        "# numeric columns and hash-bucket/embedding columns for categorical features.\n",
        "def create_feature_columns(columns, feature_spec):\n",
        "    ret = []\n",
        "    for col in columns:\n",
        "        if feature_spec[col].dtype is tf.int64 or feature_spec[col].dtype is tf.float32:\n",
        "            ret.append(tf.feature_column.numeric_column(col))\n",
        "        else:\n",
        "            ret.append(tf.feature_column.indicator_column(\n",
        "                tf.feature_column.categorical_column_with_vocabulary_list(col, list(df[col].unique()))))\n",
        "    return ret\n",
        "\n",
        "# An input function for providing input to a model from tf.Examples\n",
        "def tfexamples_input_fn(examples, feature_spec, label, mode=tf.estimator.ModeKeys.EVAL,\n",
        "                       num_epochs=None, \n",
        "                       batch_size=64):\n",
        "    def ex_generator():\n",
        "        for i in range(len(examples)):\n",
        "            yield examples[i].SerializeToString()\n",
        "    dataset = tf.data.Dataset.from_generator(\n",
        "      ex_generator, tf.dtypes.string, tf.TensorShape([]))\n",
        "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)\n",
        "    dataset = dataset.batch(batch_size)\n",
        "    dataset = dataset.map(lambda tf_example: parse_tf_example(tf_example, label, feature_spec))\n",
        "    dataset = dataset.repeat(num_epochs)\n",
        "    return dataset\n",
        "\n",
        "# Parses Tf.Example protos into features for the input function.\n",
        "def parse_tf_example(example_proto, label, feature_spec):\n",
        "    parsed_features = tf.io.parse_example(serialized=example_proto, features=feature_spec)\n",
        "    target = parsed_features.pop(label)\n",
        "    return parsed_features, target\n",
        "\n",
        "# Converts a dataframe into a list of tf.Example protos.\n",
        "def df_to_examples(df, columns=None):\n",
        "    examples = []\n",
        "    if columns == None:\n",
        "        columns = df.columns.values.tolist()\n",
        "    for index, row in df.iterrows():\n",
        "        example = tf.train.Example()\n",
        "        for col in columns:\n",
        "            if df[col].dtype is np.dtype(np.int64):\n",
        "                example.features.feature[col].int64_list.value.append(int(row[col]))\n",
        "            elif df[col].dtype is np.dtype(np.float64):\n",
        "                example.features.feature[col].float_list.value.append(row[col])\n",
        "            elif row[col] == row[col]:\n",
        "                example.features.feature[col].bytes_list.value.append(row[col].encode('utf-8'))\n",
        "        examples.append(example)\n",
        "    return examples\n",
        "\n",
        "# Converts a dataframe column into a column of 0's and 1's based on the provided test.\n",
        "# Used to force label columns to be numeric for binary classification using a TF estimator.\n",
        "def make_label_column_numeric(df, label_column, test):\n",
        "  df[label_column] = np.where(test(df[label_column]), 1, 0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nu398ARdeuxe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Read training dataset from CSV {display-mode: \"form\"}\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "# Set the path to the CSV containing the dataset to train on.\n",
        "csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'\n",
        "\n",
        "# Set the column names for the columns in the CSV. If the CSV's first line is a header line containing\n",
        "# the column names, then set this to None.\n",
        "csv_columns = [\n",
        "  \"Age\", \"Workclass\", \"fnlwgt\", \"Education\", \"Education-Num\", \"Marital-Status\",\n",
        "  \"Occupation\", \"Relationship\", \"Race\", \"Sex\", \"Capital-Gain\", \"Capital-Loss\",\n",
        "  \"Hours-per-week\", \"Country\", \"Over-50K\"]\n",
        "\n",
        "# Read the dataset from the provided CSV and print out information about it.\n",
        "df = pd.read_csv(csv_path, names=csv_columns, skipinitialspace=True)\n",
        "\n",
        "df"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "67DYIFxoevt2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Specify input columns and column to predict {display-mode: \"form\"}\n",
        "import numpy as np\n",
        "\n",
        "# Set the column in the dataset you wish for the model to predict\n",
        "label_column = 'Over-50K'\n",
        "\n",
        "# Make the label column numeric (0 and 1), for use in our model.\n",
        "# In this case, examples with a target value of '>50K' are considered to be in\n",
        "# the '1' (positive) class and all other examples are considered to be in the\n",
        "# '0' (negative) class.\n",
        "make_label_column_numeric(df, label_column, lambda val: val == '>50K')\n",
        "\n",
        "# Set list of all columns from the dataset we will use for model input.\n",
        "input_features = [\n",
        "  'Age', 'Workclass', 'Education', 'Marital-Status', 'Occupation',\n",
        "  'Relationship', 'Race', 'Sex', 'Capital-Gain', 'Capital-Loss',\n",
        "  'Hours-per-week', 'Country']\n",
        "\n",
        "# Create a list containing all input features and the label column\n",
        "features_and_labels = input_features + [label_column]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BV4f_4_Lex22",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Convert dataset to tf.Example protos {display-mode: \"form\"}\n",
        "\n",
        "examples = df_to_examples(df)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YyLr-_0de1Ii",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Create and train the linear classifier {display-mode: \"form\"}\n",
        "\n",
        "num_steps = 2000  #@param {type: \"number\"}\n",
        "\n",
        "# Create a feature spec for the classifier\n",
        "feature_spec = create_feature_spec(df, features_and_labels)\n",
        "\n",
        "# Define and train the classifier\n",
        "train_inpf = functools.partial(tfexamples_input_fn, examples, feature_spec, label_column)\n",
        "classifier = tf.estimator.LinearClassifier(\n",
        "    feature_columns=create_feature_columns(input_features, feature_spec))\n",
        "classifier.train(train_inpf, steps=num_steps)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EbZuyaJsufkz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Create and train the DNN classifier {display-mode: \"form\"}\n",
        "num_steps_2 = 2000  #@param {type: \"number\"}\n",
        "\n",
        "classifier2 = tf.estimator.DNNClassifier(\n",
        "    feature_columns=create_feature_columns(input_features, feature_spec),\n",
        "    hidden_units=[128, 64, 32])\n",
        "classifier2.train(train_inpf, steps=num_steps_2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NUQVro76e38Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Invoke What-If Tool for test data and the trained models {display-mode: \"form\"}\n",
        "\n",
        "num_datapoints = 2000  #@param {type: \"number\"}\n",
        "tool_height_in_px = 1000  #@param {type: \"number\"}\n",
        "\n",
        "from witwidget.notebook.visualization import WitConfigBuilder\n",
        "from witwidget.notebook.visualization import WitWidget\n",
        "\n",
        "# Load up the test dataset\n",
        "test_csv_path = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test'\n",
        "test_df = pd.read_csv(test_csv_path, names=csv_columns, skipinitialspace=True,\n",
        "  skiprows=1)\n",
        "make_label_column_numeric(test_df, label_column, lambda val: val == '>50K.')\n",
        "test_examples = df_to_examples(test_df[0:num_datapoints])\n",
        "\n",
        "# Setup the tool with the test examples and the trained classifier\n",
        "config_builder = WitConfigBuilder(test_examples[0:num_datapoints]).set_estimator_and_feature_spec(\n",
        "    classifier, feature_spec).set_compare_estimator_and_feature_spec(\n",
        "    classifier2, feature_spec).set_label_vocab(['Under 50K', 'Over 50K'])\n",
        "a = WitWidget(config_builder, height=tool_height_in_px)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wez70n72Q5lM",
        "colab_type": "text"
      },
      "source": [
        "#### Exploration ideas\n",
        "\n",
        "- Organize datapoints by setting X-axis scatter to \"inference score 1\" and Y-axis scatter to \"inference score 2\" to see how each datapoint differs in score between the linear model (1) and DNN model (2). Points off the diagonal have differences in results between the two models.\n",
        "  - Are there patterns of which datapoints don't agree between the two models?\n",
        "  - If you set the ground truth feature dropdown in the \"Performance + Fairness\" tab to \"Over-50K\", then you can color or bin the datapoints by \"inference correct 1\" or \"inference correct 2\". Are there patterns of which datapoints are incorrect for model 1? For model 2?\n",
        "\n",
        "- Explore performance of the two models through the confusion matrices in the \"Performance + Fairness\" tab. Which model is best? Train either model for longer and see if you can change this. Are the rates of errors (false positives and false negatives) that the two models make different?\n",
        "  - Click the \"optimize threshold\" button to set the optimal positive classification threshold for each model based on the current cost ratio of 1. How do those thresholds and the resulting confusion matrices differ?\n",
        "    - Change the cost ratio and optimize the threshold again. How does the threshold and performance change on the two models?\n",
        "  - Slice the dataset by features, such as \"sex\" or \"race\". Does either model have more-equal performance between slices?\n",
        "    - Use the threshold optimization buttons to set optimal thresholds based on the different fairness constraints. How does performance between slices differ between the two models. Does one require larger differences in threshold values per slice to achieve the desired constraint?\n",
        "\n",
        "- Looking at the create_feature_columns function in the \"Define helper methods\" cell, categorical features use one-hot encodings in the model. Perhaps change a many-valued categorical feature, such as education to use an embedding layer. Does anything change in the model behavior (can look through partial dependence plots as one way to investigate)."
      ]
    }
  ]
}