sentiment_analysis_unified_information_explainer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Copyright (c) Microsoft Corporation. All rights reserved.*\n",
    "\n",
    "*Licensed under the MIT License.*\n",
    "\n",
    "# Sentiment Analysis and the Unified Information Explainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Before You Start\n",
    "\n",
    "> **Tip**: If you want to run through the notebook quickly, you can set the **`QUICK_RUN`** flag in the cell below to **`True`**. This will run the notebook on a small subset of the data and a use a smaller number of epochs. \n",
    "\n",
    "If you run into CUDA out-of-memory error or the jupyter kernel dies constantly, try reducing the `BATCH_SIZE` and `MAX_LEN`, but note that model performance will be compromised. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Set QUICK_RUN = True to run the notebook on a small subset of data and a smaller number of epochs.\n",
    "QUICK_RUN = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import os\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import nlp\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "from interpret_text.experimental.common.utils_bert import Language, Tokenizer, BERTSequenceClassifier\n",
    "from interpret_text.experimental.unified_information import UnifiedInformationExplainer\n",
    "from interpret_text.experimental.common.timer import Timer\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "In this notebook, we fine-tune and evaluate a pretrained [BERT](https://arxiv.org/abs/1810.04805) model on a subset of the [Emo](https://huggingface.co/datasets/emo/) dataset.\n",
    "\n",
    "We use a [sequence classifier](https://github.com/microsoft/nlp/blob/master/utils_nlp/models/bert/sequence_classification.py) that wraps [Hugging Face's PyTorch implementation](https://github.com/huggingface/pytorch-pretrained-BERT) of Google's [BERT](https://github.com/google-research/bert).\n",
    "\n",
    "We then show how to use the [interpret-text](https://github.com/interpretml/interpret-text) package to explain the outcome of the model. More example notebooks and NLP explainer models can be found on the [github page](https://github.com/interpretml/interpret-text)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set parameters\n",
    "First we set some parameters that we use for our modeling task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEST_DATA_FRACTION = 1\n",
    "NUM_EPOCHS = 1\n",
    "\n",
    "if QUICK_RUN:\n",
    "    TRAIN_DATA_FRACTION = 0.01\n",
    "    TEST_DATA_FRACTION = 0.01\n",
    "    NUM_EPOCHS = 1\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    BATCH_SIZE = 1\n",
    "else:\n",
    "    BATCH_SIZE = 2\n",
    "\n",
    "DATA_FOLDER = \"./temp\"\n",
    "BERT_CACHE_DIR = \"./temp\"\n",
    "LANGUAGE = Language.ENGLISH\n",
    "TO_LOWER = True\n",
    "MAX_LEN = 150\n",
    "BATCH_SIZE_PRED = 512\n",
    "TRAIN_SIZE = 0.6\n",
    "EMOTION_COL = \"emotion\"\n",
    "LABEL_COL = \"label\"\n",
    "TEXT_COL = \"text\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Dataset\n",
    "We then load the data.\n",
    "\n",
    "In this dataset, given a textual dialogue i.e. an utterance along with two previous turns of context, the goal was to infer the underlying emotion of the utterance by choosing from four emotion classes - Happy, Sad, Angry and Others.\n",
    "\n",
    "The Emo dataset is mainly used for sentiment analysis task, where the input is a textual dialogue and the labels are underlying emtion of the dialogue. Given the input, the goal is to identify this underlying emotion from four classes - *happy, sad, angry and others.*\n",
    "\n",
    "For our classification task, we use three of the classes - happy, sad, angry. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train, test = nlp.load_dataset(\"emo\", split = [\"train\", \"test\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2label = {0: 'others', 1: 'happy', 2: 'sad', 3: 'angry'}\n",
    "labels=list(id2label.values())\n",
    "label2id = {}\n",
    "for i,label in enumerate(labels):\n",
    "    label2id[label]=i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train_data={TEXT_COL:[],\n",
    "     EMOTION_COL:[]}\n",
    "for val in train:\n",
    "    if id2label[val[LABEL_COL]]!='others':\n",
    "        train_data[TEXT_COL].append(val[TEXT_COL])\n",
    "        train_data[EMOTION_COL].append(id2label[val[LABEL_COL]])\n",
    "        \n",
    "train_data = pd.DataFrame(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data={TEXT_COL:[],\n",
    "     EMOTION_COL:[]}\n",
    "for val in test:\n",
    "    if id2label[val[LABEL_COL]]!='others':\n",
    "        test_data[TEXT_COL].append(val[TEXT_COL])\n",
    "        test_data[EMOTION_COL].append(id2label[val[LABEL_COL]])\n",
    "        \n",
    "test_data = pd.DataFrame(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Encode labels\n",
    "label_encoder = LabelEncoder()\n",
    "train_data[LABEL_COL] = label_encoder.fit_transform(train_data[EMOTION_COL])\n",
    "test_data[LABEL_COL] = label_encoder.transform(test_data[EMOTION_COL])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution of Emotion labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset has three emotions. Here is the distribution in the training and test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = train_data.emotion.value_counts().plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = test_data.emotion.value_counts().plot.bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data[EMOTION_COL].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data[EMOTION_COL].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of training examples: {}\".format(train_data.shape[0]))\n",
    "print(\"Number of testing examples: {}\".format(test_data.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize and Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start training, we tokenize the text documents and convert them to lists of tokens. The following steps instantiate a `BERT tokenizer` given the language, and tokenize the text of the training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cpu\")# if not torch.cuda.is_available() else \"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(LANGUAGE, to_lower=TO_LOWER, cache_dir=BERT_CACHE_DIR)\n",
    "\n",
    "tokens_train = tokenizer.tokenize(list(train_data[TEXT_COL]))\n",
    "tokens_test = tokenizer.tokenize(list(test_data[TEXT_COL]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, we perform the following preprocessing steps in the cell below:\n",
    "- Convert the tokens into token indices corresponding to the BERT tokenizer's vocabulary\n",
    "- Add the special tokens [CLS] and [SEP] to mark the beginning and end of a sentence, respectively\n",
    "- Pad or truncate the token lists to the specified max length. In this case, `MAX_LEN = 150`\n",
    "- Return mask lists that indicate the paddings' positions\n",
    "- Return token type id lists that indicate which sentence the tokens belong to (not needed for one-sequence classification)\n",
    "\n",
    "*See the original [implementation](https://github.com/google-research/bert/blob/master/run_classifier.py) for more information on BERT's input format.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_train, mask_train, _ = tokenizer.preprocess_classification_tokens(tokens_train, MAX_LEN)\n",
    "tokens_test, mask_test, _ = tokenizer.preprocess_classification_tokens(tokens_test, MAX_LEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence Classifier Model\n",
    "Next, we use a sequence classifier that loads a pre-trained BERT model, given the language and number of labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = BERTSequenceClassifier(language=LANGUAGE, num_labels=3, cache_dir=BERT_CACHE_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier.model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train Model\n",
    "We train the classifier using the training set. This involves fine-tuning the BERT Transformer and learning a linear classification layer on top of that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with Timer() as t:\n",
    "    classifier.fit(token_ids=tokens_train,\n",
    "                    input_mask=mask_train,\n",
    "                    labels=train_data[LABEL_COL],    \n",
    "                    num_epochs=NUM_EPOCHS,\n",
    "                    batch_size=BATCH_SIZE,    \n",
    "                    verbose=True)\n",
    "   \n",
    "print(\"[Training time: {:.3f} hrs]\".format(t.interval / 3600))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score Model\n",
    "We score the test set using the trained classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = classifier.predict(token_ids=tokens_test, \n",
    "                           input_mask=mask_test, \n",
    "                           batch_size=BATCH_SIZE_PRED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Model\n",
    "Finally, we compute the overall accuracy, precision, recall, and F1 metrics on the test set. We also look at the metrics for eact of the genres in the the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "report = classification_report(test_data[LABEL_COL], preds, target_names=label_encoder.classes_, output_dict=True) \n",
    "accuracy = accuracy_score(test_data[LABEL_COL], preds)\n",
    "print(\"accuracy: {}\".format(accuracy))\n",
    "print(json.dumps(report, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explain Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cpu\")# if not torch.cuda.is_available() else \"cuda\")\n",
    "#download the trained model\n",
    "classifier.model.to(device)\n",
    "for param in classifier.model.parameters():\n",
    "    param.requires_grad = False\n",
    "classifier.model.eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interpreter_unified = UnifiedInformationExplainer(model=classifier.model, \n",
    "                                 train_dataset=list(train_data[TEXT_COL]), \n",
    "                                 device=device, \n",
    "                                 target_layer=14,\n",
    "                                classes=label_encoder.classes_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "idx = 3\n",
    "text = test_data[TEXT_COL][idx]\n",
    "true_label = label_encoder.inverse_transform([test_data[LABEL_COL][idx]])\n",
    "predicted_label = label_encoder.inverse_transform([preds[idx]])\n",
    "print(text, '; true label:', true_label,'; predicted label:', predicted_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation_unified = interpreter_unified.explain_local(text, 'happy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from interpret_text.experimental.widget import ExplanationDashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ExplanationDashboard(explanation_unified)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}