C3_W4_Lab_1_TFMA.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "C3_W4_Lab_1_TFMA.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/defreeze/Machine-learning-for-production-course/blob/main/C3_W4_Lab_1_TFMA.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YuSYVbwEYNHw"
      },
      "source": [
        "# Ungraded Lab: TensorFlow Model Analysis\n",
        "\n",
        "In production systems, the decision to deploy a model usually goes beyond the global metrics (e.g. accuracy) set during training. It is also important to evaluate how your model performs in different scenarios. For instance, does your weather forecasting model perform equally well in summer compared to winter? Or does your camera-based defect detector work only in certain lighting conditions? This type of investigation helps to ensure that your model can handle different cases. More than that, it can help uncover any learned biases that can result in a negative experience for your users. For example, if you're supposed to have a gender-neutral application, you don't want your model to only work well for one while poorly for another.\n",
        "\n",
        "In this lab, you will be working with [TensorFlow Model Analysis (TFMA)](https://www.tensorflow.org/tfx/guide/tfma) -- a library built specifically for analyzing a model's performance across different configurations. It allows you to specify slices of your data, then it will compute and visualize how your model performs on each slice. You can also set thresholds that your model must meet before it is marked ready for deployment. These help you make better decisions regarding any improvements you may want to make to boost your model's performance and ensure fairness.\n",
        "\n",
        "For this exercise, you will use TFMA to analyze models trained on the [Census Income dataset](http://archive.ics.uci.edu/ml/datasets/Census+Income). Specifically, you will:\n",
        "\n",
        "* study and setup the starter files to use with TFMA\n",
        "* make a configuration file to tell TFMA what data slices it will analyze and the metrics it will compute\n",
        "* visualize TFMA's outputs in a notebook environment\n",
        "* generate a time series of a model's performance\n",
        "* compare the performance of two models so you can decide which one to push to production\n",
        "\n",
        "*Credits: Some of the code and discussions are based on the TensorFlow team's [official tutorial](https://www.tensorflow.org/tfx/tutorials/model_analysis/tfma_basic).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VUHcOxH0W2PH"
      },
      "source": [
        "## Setup\n",
        "\n",
        "In this section, you will first setup your workspace to have all the modules and files to work with TFMA. You will \n",
        "* install required libraries, \n",
        "* download starter files that will contain the dataset, schema, and pretrained models you will analyze\n",
        "* prepare the dataset so it can be consumed by TFMA\n",
        "* observe how the models transform the raw features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q7-ouHFnWAsu"
      },
      "source": [
        "### Install Jupyter Extensions\n",
        "If running in a local Jupyter notebook, then these Jupyter extensions must be installed in the environment before running Jupyter. These are already available in Colab so we'll just leave the commands here for reference.\n",
        "\n",
        "```bash\n",
        "jupyter nbextension enable --py widgetsnbextension --sys-prefix \n",
        "jupyter nbextension install --py --symlink tensorflow_model_analysis --sys-prefix \n",
        "jupyter nbextension enable --py tensorflow_model_analysis --sys-prefix \n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LZj-impiAD_l"
      },
      "source": [
        "### Install libraries\n",
        "\n",
        "This will pull in all the dependencies, and will take a minute.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8X32Q_lIKYxH"
      },
      "source": [
        "# Upgrade pip to the latest version and install required packages\n",
        "!pip install -U pip\n",
        "!pip install tensorflow==2.6.0\n",
        "!pip install tensorflow_data_validation==1.1.0\n",
        "!pip install tensorflow-transform==1.0.0\n",
        "!pip install tensorflow-model-analysis==0.32.0\n",
        "!pip install apache-beam==2.32.0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7121_u1LO5W"
      },
      "source": [
        "*Note: In Google Colab, you need to restart the runtime at this point to finalize updating the packages you just installed. **Please do not proceed to the next section without restarting.** You can also ignore the errors about version incompatibility of some of the bundled packages because we won't be using those in this notebook.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "saT03A81Z4Nr"
      },
      "source": [
        "### Check the installation\n",
        "\n",
        "Running the code below should show the versions of the packages. Please re-run the install if you are seeing errors and don't forget to restart the runtime after re-installation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SA2E343NAMRF"
      },
      "source": [
        "# Import packages and print versions\n",
        "import tensorflow as tf\n",
        "import tensorflow_model_analysis as tfma\n",
        "import tensorflow_data_validation as tfdv\n",
        "\n",
        "print('TF version: {}'.format(tf.__version__))\n",
        "print('TFMA version: {}'.format(tfma.__version__))\n",
        "print('TFDV version: {}'.format(tfdv.__version__))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RptgLn2RYuK3"
      },
      "source": [
        "### Load the files\n",
        "\n",
        "Next, you will download the files you will need for this exercise:\n",
        "\n",
        "* Test datasets\n",
        "* Data schema\n",
        "* Pretrained models\n",
        "\n",
        "We've also defined some global variables below so you can access these files throughout the notebook more easily."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K4QXVIM7iglN"
      },
      "source": [
        "import os\n",
        "\n",
        "# String variables for file and directory names\n",
        "URL = 'https://storage.googleapis.com/mlep-public/course_3/week4/C3_W4_Lab_1_starter_files.tar.gz'\n",
        "TAR_NAME = 'C3_W4_Lab_1_starter_files.tar.gz'\n",
        "BASE_DIR = 'starter_files'\n",
        "DATA_DIR = os.path.join(BASE_DIR, 'data')\n",
        "CSV_DIR = os.path.join(DATA_DIR, 'csv')\n",
        "TFRECORD_DIR = os.path.join(DATA_DIR, 'tfrecord')\n",
        "MODELS_DIR = os.path.join(BASE_DIR, 'models')\n",
        "SCHEMA_FILE = os.path.join(BASE_DIR, 'schema.pbtxt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mw6d-3aNw-fm"
      },
      "source": [
        "# uncomment this line if you've downloaded the files before and want to reset\n",
        "# !rm -rf {BASE_DIR}\n",
        "\n",
        "# Download the tar file from GCP\n",
        "!wget {URL}\n",
        "\n",
        "# Extract the tar file to the base directory\n",
        "!tar xzf {TAR_NAME}\n",
        "\n",
        "# Delete tar file\n",
        "!rm {TAR_NAME}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wrO1e6QnVf9I"
      },
      "source": [
        "You can see the top level file and directories by running the cell below (or just using the file explorer on the left side of this Colab). We'll discuss what each contain in the next sections."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NpNZrJl-YJq9"
      },
      "source": [
        "print(\"Here's what we downloaded:\")\n",
        "!ls {BASE_DIR}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UBhvex2jSBa_"
      },
      "source": [
        "### Preview the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iIxhCtvaV0cC"
      },
      "source": [
        "The `data/csv` directory contains the test split of the Census Income dataset. We've divided it into several files for this demo notebook:\n",
        "\n",
        "* `data_test.csv` - 15000 rows of test data\n",
        "* `data_test_1.csv` - first 5000 rows of `data_test.csv`\n",
        "* `data_test_2.csv` - next 5000 rows of `data_test.csv`\n",
        "* `data_test_3.csv` - last 5000 rows of `data_test.csv`\n",
        "\n",
        "You can see the description of each column [here](http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.names) (please open link in a new window if Colab prevents the download). Also for simplicity, we've already preprocessed the `label` column as binary (i.e. `0` or `1`) to match the model's output. In your own projects, your labels might be in a different data type (e.g. string) and you want to transform that first so you can evaluate your model properly. You can preview the first few rows below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C4LCdt9zQ1oL"
      },
      "source": [
        "# Path to the full test set\n",
        "TEST_DATA_PATH = os.path.join(CSV_DIR, 'data_test.csv')\n",
        "\n",
        "# Preview the first few rows\n",
        "!head {TEST_DATA_PATH}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_xa7ZDV1MycO"
      },
      "source": [
        "### Parse the Schema\n",
        "\n",
        "You also downloaded a schema generated by [TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/). You should be familiar with this file type already from previous courses. You will load it now so you can use it in the later parts of the notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYo2SHFzf6_t"
      },
      "source": [
        "# Load the schema as a protocol buffer\n",
        "SCHEMA = tfdv.load_schema_text(SCHEMA_FILE)\n",
        "\n",
        "# Display the schema\n",
        "tfdv.display_schema(SCHEMA)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UP3yuJxfNXRL"
      },
      "source": [
        "### Use the Schema to Create TFRecords\n",
        "\n",
        "TFMA needs a TFRecord file input so you need to convert the CSVs in the data directory. If you've done the earlier labs, you will know that this can be easily done with `ExampleGen`. For this notebook however, you will use the helper function below instead to demonstrate how it can be done outside a TFX pipeline. You will pass in the schema you loaded in the previous step to determine the correct type of each feature."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8-wud3fPczl6"
      },
      "source": [
        "# imports for helper function\n",
        "import csv\n",
        "from tensorflow.core.example import example_pb2\n",
        "from tensorflow_metadata.proto.v0 import schema_pb2\n",
        "\n",
        "def csv_to_tfrecord(schema, csv_file, tfrecord_file):\n",
        "  ''' Converts a csv file into a tfrecord\n",
        "  Args:\n",
        "    schema (schema_pb2) - Schema protobuf from TFDV\n",
        "    csv_file (string) - file to convert to tfrecord\n",
        "    tfrecord_file (string) - filename of tfrecord to create\n",
        "\n",
        "  Returns:\n",
        "    filename of tfrecord\n",
        "  '''\n",
        "\n",
        "  # Open CSV file for reading. Each row is mapped as a dictionary.\n",
        "  reader = csv.DictReader(open(csv_file, 'r'))\n",
        "  \n",
        "  # Initialize TF examples list\n",
        "  examples = []\n",
        "\n",
        "  # For each row in CSV, create a TF Example based on\n",
        "  # the Schema and append to the list\n",
        "  for line in reader:\n",
        "\n",
        "    # Intialize example\n",
        "    example = example_pb2.Example()\n",
        "\n",
        "    # Loop through features in the schema\n",
        "    for feature in schema.feature:\n",
        "\n",
        "      # Get current feature name\n",
        "      key = feature.name\n",
        "\n",
        "      # Populate values based on data type of current feature\n",
        "      if feature.type == schema_pb2.FLOAT:\n",
        "        example.features.feature[key].float_list.value[:] = (\n",
        "            [float(line[key])] if len(line[key]) > 0 else [])\n",
        "      elif feature.type == schema_pb2.INT:\n",
        "        example.features.feature[key].int64_list.value[:] = (\n",
        "            [int(line[key])] if len(line[key]) > 0 else [])\n",
        "      elif feature.type == schema_pb2.BYTES:\n",
        "        example.features.feature[key].bytes_list.value[:] = (\n",
        "            [line[key].encode('utf8')] if len(line[key]) > 0 else [])\n",
        "        \n",
        "    # Append to the list\n",
        "    examples.append(example)\n",
        "\n",
        "  # Write examples to tfrecord file\n",
        "  with tf.io.TFRecordWriter(tfrecord_file) as writer:\n",
        "    for example in examples:\n",
        "      writer.write(example.SerializeToString())\n",
        "  \n",
        "  return tfrecord_file"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sSO1fOrrYZjK"
      },
      "source": [
        "The code below will do the conversion and we've defined some more global variables that you will use in later exercises. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xU8R41EexIoc"
      },
      "source": [
        "# Create tfrecord directory\n",
        "!mkdir {TFRECORD_DIR}\n",
        "\n",
        "# Create list of tfrecord files\n",
        "tfrecord_files = [csv_to_tfrecord(SCHEMA, f'{CSV_DIR}/{name}', f\"{TFRECORD_DIR}/{name.replace('csv','tfrecord')}\") \n",
        "  for name in os.listdir(CSV_DIR)]\n",
        "\n",
        "# Print created files\n",
        "print(f'files created: {tfrecord_files}')\n",
        "\n",
        "# Create variables for each tfrecord\n",
        "TFRECORD_FULL = os.path.join(TFRECORD_DIR, 'data_test.tfrecord')\n",
        "TFRECORD_DAY1 = os.path.join(TFRECORD_DIR, 'data_test_1.tfrecord')\n",
        "TFRECORD_DAY2 = os.path.join(TFRECORD_DIR, 'data_test_2.tfrecord')\n",
        "TFRECORD_DAY3 = os.path.join(TFRECORD_DIR, 'data_test_3.tfrecord')\n",
        "\n",
        "# Delete unneeded variable\n",
        "del tfrecord_files"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wr6d8JqzR8TZ"
      },
      "source": [
        "### Pretrained models\n",
        "\n",
        "Lastly, you also downloaded pretrained Keras models and they are stored in the `models/` directory. TFMA supports a number of different model types including TF Keras models, models based on generic TF2 signature APIs, as well TF estimator based models. The [get_started](https://www.tensorflow.org/tfx/model_analysis/get_started) guide has the full list of model types supported and any restrictions. You can also consult the [FAQ](https://www.tensorflow.org/tfx/model_analysis/faq) for examples on how to configure these models.\n",
        "\n",
        "We have included three models and you can choose to analyze any one of them in the later sections. These were saved in [SavedModel](https://www.tensorflow.org/guide/saved_model) format which is the default when saving with the Keras Models API."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HQ4mjuMkR7z2"
      },
      "source": [
        "# list model directories\n",
        "!ls {MODELS_DIR}\n",
        "\n",
        "# Create string variables for each model directory\n",
        "MODEL1_FILE = os.path.join(MODELS_DIR, 'model1')\n",
        "MODEL2_FILE = os.path.join(MODELS_DIR, 'model2')\n",
        "MODEL3_FILE = os.path.join(MODELS_DIR, 'model3')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0eQ0M7BI6IG"
      },
      "source": [
        "As mentioned earlier, these models were trained on the [Census Income dataset](http://archive.ics.uci.edu/ml/datasets/Census+Income). The label is `1` if a person earns more than 50k USD and `0` if less than or equal. You can load one of the models and look at the summary to get a sense of its architecture. All three models use the same architecture but were trained with different epochs to simulate varying performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vvIEhxeMxekr"
      },
      "source": [
        "# Load model 1\n",
        "model = tf.keras.models.load_model(MODEL1_FILE)\n",
        "\n",
        "# Print summary. You can ignore the warnings at the start.\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L8P-3h-HLPtH"
      },
      "source": [
        "You can see the code to build these in the next lab. For now, you'll only need to take note of a few things. First, the output is a single dense unit with a sigmoid activation (i.e. `dense_5` above). This is standard for binary classification problems.\n",
        "\n",
        "Another is that the model is exported with a transformation layer. You can see this in the summary above at the bottom row named `transform_features_layer` and it is not connected to the other layers. From previous labs, you will know that this is taken from the graph generated by the `Transform` component. It helps to avoid training-serving skews by making sure that raw inputs are transformed in the same way that the model expects. It is also available as a `tft_layer` property of the model object."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qjEsamx6620d"
      },
      "source": [
        "# Transformation layer can be accessed in two ways. These are equivalent.\n",
        "model.get_layer('transform_features_layer') is model.tft_layer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cvzrnszr9YL1"
      },
      "source": [
        "TFMA invokes this layer automatically for your raw inputs but we've included a short snippet below to demonstrate how the transformation works. You can see that the raw features are indeed reformatted to an acceptable input for the model. The raw numeric features are scaled and the raw categorical (string) features are encoded to one-hot vectors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pIUo4v_NFBuO"
      },
      "source": [
        "from tensorflow_transform.tf_metadata import schema_utils\n",
        "\n",
        "# Load one tfrecord\n",
        "tfrecord_file = tf.data.TFRecordDataset(TFRECORD_DAY1)\n",
        "\n",
        "# Parse schema object as a feature spec\n",
        "feature_spec = schema_utils.schema_as_feature_spec(SCHEMA).feature_spec\n",
        "\n",
        "# Create a batch from the dataset\n",
        "for records in tfrecord_file.batch(1).take(1):\n",
        "\n",
        "  # Parse the batch to get a dictionary of raw features\n",
        "  parsed_examples = tf.io.parse_example(records, feature_spec)\n",
        "\n",
        "  # Print the results\n",
        "  print(\"\\nRAW FEATURES:\")\n",
        "  for key, value in parsed_examples.items():\n",
        "    print(f'{key}: {value.numpy()}')\n",
        "  \n",
        "  # Pop the label since the model does not expect a label input\n",
        "  parsed_examples.pop('label')\n",
        "\n",
        "  # Transform the rest of the raw features using the transform layer\n",
        "  transformed_examples = model.tft_layer(parsed_examples)\n",
        "\n",
        "  # Print the input to the model\n",
        "  print(\"\\nTRANSFORMED FEATURES:\")\n",
        "  for key, value in transformed_examples.items():\n",
        "    print(f'{key}: {value.numpy()}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEeCIAwXJnDr"
      },
      "source": [
        "The transformed features can be passed into the model to get the predictions. The snippet below demonstrates this and we used a low-threshold [BinaryAccuracy](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/BinaryAccuracy) metric to compare the true labels and model predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FXIrFLKPyHtz"
      },
      "source": [
        "from tensorflow_transform.tf_metadata import schema_utils\n",
        "\n",
        "# Load one tfrecord\n",
        "tfrecord_file = tf.data.TFRecordDataset(TFRECORD_DAY1)\n",
        "\n",
        "# Parse schema object as a feature spec\n",
        "feature_spec = schema_utils.schema_as_feature_spec(SCHEMA).feature_spec\n",
        "\n",
        "# Create a batch from the dataset\n",
        "for records in tfrecord_file.batch(5).take(1):\n",
        "\n",
        "  # Get the label values from the raw input\n",
        "  parsed_examples = tf.io.parse_example(records, feature_spec)\n",
        "  y_true = parsed_examples.pop('label')\n",
        "  print(f'labels:\\n {y_true.numpy()}\\n')\n",
        "  \n",
        "  # Transform the raw features and pass to the model to get predictions\n",
        "  transformed_examples = model.tft_layer(parsed_examples)\n",
        "  y_pred = model(transformed_examples)\n",
        "  print(f'predictions:\\n {y_pred.numpy()}\\n')\n",
        "  \n",
        "  # Measure the binary accuracy\n",
        "  metric = tf.keras.metrics.BinaryAccuracy(threshold=0.3)\n",
        "  metric.update_state(y_true, y_pred)\n",
        "  print(f'binary accuracy: {metric.result().numpy()}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGFEc_P0LH8b"
      },
      "source": [
        "Last thing to note is the model is also exported with a [serving signature](https://www.tensorflow.org/guide/saved_model#specifying_signatures_during_export). You will know more about this in the next lab and in later parts of the specialization but for now, you can think of it as a configuration for when the model is deployed for inference. For this particular model, the default signature is configured to transform batches of serialized raw features before feeding to the model input. That way, you wouldn't have to explicitly code the transformations as previously shown in the snippet above. You can just pass in batches of data directly as shown below. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nh9rvKbSUW_l"
      },
      "source": [
        "# Load one tfrecord\n",
        "tfrecord_file = tf.data.TFRecordDataset(TFRECORD_DAY1)\n",
        "\n",
        "# Print available signatures\n",
        "print(f'model signatures: {model.signatures}\\n')\n",
        "\n",
        "# Create a batch\n",
        "for records in tfrecord_file.batch(5).take(1):\n",
        "\n",
        "  # Pass the batch to the model serving signature to get predictions\n",
        "  output = model.signatures['serving_default'](examples=records)\n",
        "\n",
        "  # Print results\n",
        "  print(f\"predictions:\\n {output['output_0']}\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8gNoFWd3N-9b"
      },
      "source": [
        "TFMA accesses this model signature so it can work with the raw data and evaluate the model to get the metrics. Not only that, it can also extract specific features and domain values from your dataset before it computes these metrics. Let's see how this is done in the next section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fp8Ub7GTXH3j"
      },
      "source": [
        "## Setup and Run TFMA\n",
        "\n",
        "With the dataset and model now available, you can now move on to use TFMA. There are some additional steps needed:\n",
        "\n",
        "* Create a `tfma.EvalConfig` protocol message containing details about the models, metrics, and data slices you'd like to analyze\n",
        "* Create a `tfma.EvalSharedModel` that points to your saved models.\n",
        "* Specify an output path where the analysis results will be stored "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qgC7NdCatT8y"
      },
      "source": [
        "### Create EvalConfig\n",
        "\n",
        "The [tfma.EvalConfig()](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalConfig?hl=nn&skip_cache=true) is a protocol message that sets up the analysis. Here, you will specify:\n",
        "\n",
        "* [`model_specs`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/ModelSpec) - message containing at least the label key so it can be extracted from the evaluation/test data\n",
        "\n",
        "* [`metrics_specs`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/MetricsSpec) - containing the metrics you would like to evaluate. A comprehensive guide can be found [here](https://www.tensorflow.org/tfx/model_analysis/metrics) and you will use the binary classification metrics for this exercise.\n",
        "\n",
        "* [`slicing_specs`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/SlicingSpec) - containing the feature slices you would like to compute metrics for. A short description of different types of slices is shown [here](https://www.tensorflow.org/tfx/model_analysis/setup#slicing_specs)\n",
        "\n",
        "The eval config should be passed as a protocol message and you can use the [google.protobuf.text_format](https://googleapis.dev/python/protobuf/latest/google/protobuf/text_format.html) module for that as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLJxcjpjfwkx"
      },
      "source": [
        "import tensorflow_model_analysis as tfma\n",
        "from google.protobuf import text_format\n",
        "\n",
        "# Setup tfma.EvalConfig settings\n",
        "eval_config = text_format.Parse(\"\"\"\n",
        "  ## Model information\n",
        "  model_specs {\n",
        "    # For keras (and serving models), you need to add a `label_key`.\n",
        "    label_key: \"label\"\n",
        "  }\n",
        "\n",
        "  ## Post training metric information. These will be merged with any built-in\n",
        "  ## metrics from training.\n",
        "  metrics_specs {\n",
        "    metrics { class_name: \"ExampleCount\" }\n",
        "    metrics { class_name: \"BinaryAccuracy\" }\n",
        "    metrics { class_name: \"BinaryCrossentropy\" }\n",
        "    metrics { class_name: \"AUC\" }\n",
        "    metrics { class_name: \"AUCPrecisionRecall\" }\n",
        "    metrics { class_name: \"Precision\" }\n",
        "    metrics { class_name: \"Recall\" }\n",
        "    metrics { class_name: \"MeanLabel\" }\n",
        "    metrics { class_name: \"MeanPrediction\" }\n",
        "    metrics { class_name: \"Calibration\" }\n",
        "    metrics { class_name: \"CalibrationPlot\" }\n",
        "    metrics { class_name: \"ConfusionMatrixPlot\" }\n",
        "    # ... add additional metrics and plots ...\n",
        "  }\n",
        "\n",
        "  ## Slicing information\n",
        "\n",
        "  # overall slice\n",
        "  slicing_specs {}\n",
        "\n",
        "  # slice specific features\n",
        "  slicing_specs {\n",
        "    feature_keys: [\"sex\"]\n",
        "  }\n",
        "  slicing_specs {\n",
        "    feature_keys: [\"race\"]\n",
        "  }\n",
        "\n",
        "  # slice specific values from features\n",
        "  slicing_specs {\n",
        "    feature_values: {\n",
        "      key: \"native-country\"\n",
        "      value: \"Cambodia\"\n",
        "    }\n",
        "  }\n",
        "  slicing_specs {\n",
        "    feature_values: {\n",
        "      key: \"native-country\"\n",
        "      value: \"Canada\"\n",
        "    }\n",
        "  }\n",
        "\n",
        "  # slice feature crosses\n",
        "  slicing_specs {\n",
        "    feature_keys: [\"sex\", \"race\"]\n",
        "  }\n",
        "\"\"\", tfma.EvalConfig())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J165kzHNTEQN"
      },
      "source": [
        "### Create EvalSharedModel\n",
        "\n",
        "TFMA also requires an [EvalSharedModel](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/types/EvalSharedModel) instance that points to your model so it can be shared between multiple threads in the same process. This instance includes information about the type of model (keras, etc) and how to load and configure the model from its saved location on disk (e.g. tags, etc). The [tfma.default_eval_shared_model()](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/default_eval_shared_model) API can be used to create this given the model location and eval config."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KsnjbIi5OtXb"
      },
      "source": [
        "# Create a tfma.EvalSharedModel that points to your model.\n",
        "# You can ignore the warnings generated.\n",
        "eval_shared_model = tfma.default_eval_shared_model(\n",
        "    eval_saved_model_path=MODEL1_FILE,\n",
        "    eval_config=eval_config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F0s4wWgQb2LT"
      },
      "source": [
        "# Show properties of EvalSharedModel\n",
        "print(f'EvalSharedModel contents: {eval_shared_model}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pViNbIpuVYLU"
      },
      "source": [
        "### Run TFMA\n",
        "\n",
        "With the setup complete, you just need to declare an output directory then run TFMA. You will pass in the eval config, shared model, dataset, and output directory to [`tfma.run_model_analysis()`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/run_model_analysis) as shown below. This will create a [`tfma.EvalResult`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalResult) which you can use later for rendering metrics and plots."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3ej6jgiVkuY"
      },
      "source": [
        "# Specify output path for the evaluation results\n",
        "OUTPUT_DIR = os.path.join(BASE_DIR, 'output')\n",
        "\n",
        "# Run TFMA. You can ignore the warnings generated.\n",
        "eval_result = tfma.run_model_analysis(\n",
        "    eval_shared_model=eval_shared_model,\n",
        "    eval_config=eval_config,\n",
        "    data_location=TFRECORD_FULL,\n",
        "    output_path=OUTPUT_DIR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0khNBC9FlEO"
      },
      "source": [
        "## Visualizing Metrics and Plots\n",
        "\n",
        "You can visualize the results also using TFMA methods. In this section, you will view the returned metrics and plots for the different slices you specified in the eval config."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cSl9qyTCbBKR"
      },
      "source": [
        "### Rendering Metrics\n",
        "\n",
        "You can view the metrics with the [`tfma.view.render_slicing_metrics()`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/view/render_slicing_metrics) method. By default, the views will display the `Overall` slice. To view a particular slice you can pass in a feature name to the `slicing_column` argument as shown below. You can visualize the different metrics through the `Show` dropdown menu and you can hover over the bar charts to show the exact value measured. \n",
        "\n",
        "We encourage you to try the different options you see and also modify the command. Here are some examples:\n",
        "\n",
        "* Removing the `slicing_column` argument will produce the overall slice.\n",
        "* You can also pass in `race` (since it was specified in the eval config) to see the results for that particular slice.\n",
        "* Using the `Examples (Weighted) Threshold` slider above 5421 will remove the `Female` slice because it has less examples than that.\n",
        "* Toggling the `View` dropdown to `Metrics Histogram` will show the results divided into buckets. For example, if you're slicing column is `sex` and the `Histogram Type` dropdown is at `Slice Counts`, then you will one slice in two of the 10 (default) buckets since we only have two values for that feature ('Male' and 'Female'). The x-axis show the values for the metric in the `Select Metric` dropdown. This is the default view when the number of slices is large.\n",
        "* At the bottom of the screen, you will notice the measurements also presented in tabular format. You can sort it by clicking on the feature name headers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hJ5_UMnWYmaE"
      },
      "source": [
        "# Render metrics for a feature\n",
        "tfma.view.render_slicing_metrics(eval_result, slicing_column='sex')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hSnqI6Esb1XM"
      },
      "source": [
        "### More Slices\n",
        "\n",
        "If you haven't yet, you can also pass in the `native-country` to the slicing column. The difference in this visualization is we only specified two of its values in the eval config earlier. This is useful if you just want to study a subgroup of a particular feature and not the entire domain."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "355wqvY3yBod"
      },
      "source": [
        "# Render metrics for feature. Review EvalConfig message to see what values were selected.\n",
        "tfma.view.render_slicing_metrics(eval_result, slicing_column='native-country')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PsXM0NYGeajg"
      },
      "source": [
        "TFMA also supports creating feature crosses to analyze combinations of features. Our original settings created a cross between `sex` and `race` and you can pass it in as a [SlicingSpec](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/SlicingSpec) as shown below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k7vbFS1Me1SH"
      },
      "source": [
        "# Render metrics for feature crosses\n",
        "tfma.view.render_slicing_metrics(\n",
        "    eval_result,\n",
        "    slicing_spec=tfma.SlicingSpec(\n",
        "        feature_keys=['sex', 'race']))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmeODqrUfJw2"
      },
      "source": [
        "In some cases, crossing the two columns creates a lot of combinations. You can narrow down the results to only look at specific values by specifying it in the `slicing_spec` argument. Below shows the results for the `sex` feature for the `Other` race."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kdvBNfcHfRWg"
      },
      "source": [
        "# Narrow down the feature crosses by specifying feature values\n",
        "tfma.view.render_slicing_metrics(\n",
        "    eval_result,\n",
        "    slicing_spec=tfma.SlicingSpec(\n",
        "        feature_keys=['sex'], feature_values={'race': 'Other'}))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8acksU33KMm"
      },
      "source": [
        "### Rendering Plots\n",
        "\n",
        "Any plots that were added to the `tfma.EvalConfig` as post training `metric_specs` can be displayed using [`tfma.view.render_plot`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/view/render_plot).\n",
        "\n",
        "As with metrics, plots can be viewed by slice. Unlike metrics, only plots for a particular slice value can be displayed so the `tfma.SlicingSpec` must be used and it must specify both a slice feature name and value. If no slice is provided then the plots for the `Overall` slice is used.\n",
        "\n",
        "The example below displays the plots that were computed for the `sex:Male` slice. You can click on the names at the bottom of the graph to see a different plot type. Alternatively, you can tick the `Show all plots` checkbox to show all the plots in one screen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X4TCKjGw3S-a"
      },
      "source": [
        "# Render plots\n",
        "tfma.view.render_plot(\n",
        "    eval_result,\n",
        "    tfma.SlicingSpec(feature_values={'sex': 'Male'}))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meRvFkKcPbux"
      },
      "source": [
        "## Tracking Model Performance Over Time\n",
        "\n",
        "Your training dataset will be used for training your model, and will hopefully be representative of your test dataset and the data that will be sent to your model in production.  However, while the data in inference requests may remain the same as your training data, it can also start to change enough so that the performance of your model will change. That means that you need to monitor and measure your model's performance on an ongoing basis so that you can be aware of and react to changes.\n",
        "\n",
        "Let's take a look at how TFMA can help. You will load three different datasets and compare the model analysis results using the [`render_time_series()`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/view/render_time_series) method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zJYUOjmFfuPy"
      },
      "source": [
        "import os\n",
        "\n",
        "# Put data paths we prepared earlier in a list\n",
        "TFRECORDS = [TFRECORD_DAY1, TFRECORD_DAY2, TFRECORD_DAY3]\n",
        "\n",
        "# Initialize output paths list for each result \n",
        "output_paths = []\n",
        "\n",
        "# Run eval on each tfrecord separately\n",
        "for num, tfrecord in enumerate(TFRECORDS):\n",
        "\n",
        "  # Use the same model as before\n",
        "  eval_shared_model = tfma.default_eval_shared_model(\n",
        "      eval_saved_model_path=MODEL1_FILE,\n",
        "      eval_config=eval_config)\n",
        "\n",
        "  # Prepare output path name\n",
        "  output_path = os.path.join('.', 'time_series', str(num))\n",
        "  output_paths.append(output_path)\n",
        "\n",
        "  # Run TFMA on the current tfrecord in the loop\n",
        "  tfma.run_model_analysis(eval_shared_model=eval_shared_model,\n",
        "                          eval_config=eval_config,\n",
        "                          data_location=tfrecord,\n",
        "                          output_path=output_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RsO-gqCRK0ar"
      },
      "source": [
        "First, imagine that you've trained and deployed your model yesterday. And now, you want to see how it's doing on the new data coming in today.  The visualization will start by displaying AUC. From the UI, you can:\n",
        "\n",
        "* Add other metrics using the \"Add metric series\" menu.\n",
        "* Close unwanted graphs by clicking on x\n",
        "* Hover over data points (the ends of line segments in the graph) to get more details\n",
        "\n",
        "Note: In the metric series charts, the x-axis is just the model directory name of the model that you're examining."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KjEws8T0cDm9"
      },
      "source": [
        "# Load results for day 1 and day 2 datasets\n",
        "eval_results_from_disk = tfma.load_eval_results(output_paths[:2])\n",
        "\n",
        "# Visualize results\n",
        "tfma.view.render_time_series(eval_results_from_disk)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQ7kZxESN9Bx"
      },
      "source": [
        "Now imagine that another day has passed and you want to see how it's doing on the new data coming in today."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VjQmlXMmLwHf"
      },
      "source": [
        "# Load results for all three days\n",
        "eval_results_from_disk = tfma.load_eval_results(output_paths)\n",
        "\n",
        "# Visualize the results\n",
        "tfma.view.render_time_series(eval_results_from_disk)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oWD68u-emPN9"
      },
      "source": [
        "This type of investigation lets you see if your model is behaving poorly on new data. You can make the decision to retrain your production model based on these results. Retraining might not always produce the best results and you also need a way to detect that. You will see how TFMA helps in that regard in the next section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N1jpShgQxlVL"
      },
      "source": [
        "## Model Validation\n",
        "\n",
        "TFMA can be configured to evaluate multiple models at the same time. Typically, this is done to compare a candidate model against a baseline (such as the currently serving model) to determine what the performance differences in metrics are. When [thresholds](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/MetricThreshold) are configured, TFMA will produce a [`tfma.ValidationResult`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/ValidationResult) record indicating whether the performance matches expecations.\n",
        "\n",
        "Below, you will re-configure the EvalConfig settings to compare two models: a candidate and a baseline. You will also validate the candidate's performance against the baseline by setting a [`tmfa.MetricThreshold`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/MetricThreshold) on the `BinaryAccuracy` metric. This helps in determining if your new model can indeed replace your currently deployed model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kkatdR6Y1-4G"
      },
      "source": [
        "# Setup tfma.EvalConfig setting with metric thresholds\n",
        "eval_config_with_thresholds = text_format.Parse(\"\"\"\n",
        "  ## Model information\n",
        "  model_specs {\n",
        "    name: \"candidate\"\n",
        "    label_key: \"label\"\n",
        "  }\n",
        "  model_specs {\n",
        "    name: \"baseline\"\n",
        "    label_key: \"label\"\n",
        "    is_baseline: true\n",
        "  }\n",
        "\n",
        "  ## Post training metric information\n",
        "  metrics_specs {\n",
        "    metrics { class_name: \"ExampleCount\" }\n",
        "    metrics {\n",
        "      class_name: \"BinaryAccuracy\"\n",
        "      threshold {\n",
        "        # Ensure that metric is always > 0.9\n",
        "        value_threshold {\n",
        "          lower_bound { value: 0.9 }\n",
        "        }\n",
        "        # Ensure that metric does not drop by more than a small epsilon\n",
        "        # e.g. (candidate - baseline) > -1e-10 or candidate > baseline - 1e-10\n",
        "        change_threshold {\n",
        "          direction: HIGHER_IS_BETTER\n",
        "          absolute { value: -1e-10 }\n",
        "        }\n",
        "      }\n",
        "    }\n",
        "    metrics { class_name: \"BinaryCrossentropy\" }\n",
        "    metrics { class_name: \"AUC\" }\n",
        "    metrics { class_name: \"AUCPrecisionRecall\" }\n",
        "    metrics { class_name: \"Precision\" }\n",
        "    metrics { class_name: \"Recall\" }\n",
        "    metrics { class_name: \"MeanLabel\" }\n",
        "    metrics { class_name: \"MeanPrediction\" }\n",
        "    metrics { class_name: \"Calibration\" }\n",
        "    metrics { class_name: \"CalibrationPlot\" }\n",
        "    metrics { class_name: \"ConfusionMatrixPlot\" }\n",
        "    # ... add additional metrics and plots ...\n",
        "  }\n",
        "\n",
        "  ## Slicing information\n",
        "  slicing_specs {}  # overall slice\n",
        "  slicing_specs {\n",
        "    feature_keys: [\"race\"]\n",
        "  }\n",
        "  slicing_specs {\n",
        "    feature_keys: [\"sex\"]\n",
        "  }\n",
        "\"\"\", tfma.EvalConfig())\n",
        "\n",
        "# Create tfma.EvalSharedModels that points to the candidate and baseline\n",
        "candidate_model_path = MODEL1_FILE\n",
        "baseline_model_path = MODEL2_FILE\n",
        "\n",
        "eval_shared_models = [\n",
        "  tfma.default_eval_shared_model(\n",
        "      model_name=tfma.CANDIDATE_KEY,\n",
        "      eval_saved_model_path=candidate_model_path,\n",
        "      eval_config=eval_config_with_thresholds),\n",
        "  tfma.default_eval_shared_model(\n",
        "      model_name=tfma.BASELINE_KEY,\n",
        "      eval_saved_model_path=baseline_model_path,\n",
        "      eval_config=eval_config_with_thresholds),\n",
        "]\n",
        "\n",
        "# Specify validation path\n",
        "validation_output_path = os.path.join(OUTPUT_DIR, 'validation')\n",
        "\n",
        "# Run TFMA on the two models\n",
        "eval_result_with_validation = tfma.run_model_analysis(\n",
        "    eval_shared_models,\n",
        "    eval_config=eval_config_with_thresholds,\n",
        "    data_location=TFRECORD_FULL,\n",
        "    output_path=validation_output_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "siF6npd3IfJq"
      },
      "source": [
        "When running evaluations with one or more models against a baseline, TFMA automatically adds different metrics for all the metrics computed during the evaluation. These metrics are named after the corresponding metric but with the string `_diff` appended to the metric name. A positive value for these `_diff` metrics indicates an improved performance against the baseline.\n",
        "\n",
        "Like in the previous section, you can view the results with `render_time_series()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yGIw9TDuJ7wn"
      },
      "source": [
        "# Render results\n",
        "tfma.view.render_time_series(eval_result_with_validation)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JIsehm_V4oKU"
      },
      "source": [
        "You can use [`tfma.load_validator_result`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/load_validation_result) to view the validation results you specified with the threshold settings. For this example, the validation fails because `BinaryAccuracy` is below the threshold."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "48EdSTUW5eE1"
      },
      "source": [
        "# Print validation result\n",
        "validation_result = tfma.load_validation_result(validation_output_path)\n",
        "print(validation_result.validation_ok)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aBhzlF-lqRFY"
      },
      "source": [
        "**Congratulations! You have now explored the different methods of model analysis using TFMA. In the next section, you will see how these can fit into a TFX pipeline so you can automate the process and store the results in your pipeline directory and metadata store.**"
      ]
    }
  ]
}