tutorials/advanced/BRAT_Annotations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Tutorial: Creating Gold Annotation Labels with BRAT\n",
    "\n",
    "This is a short tutorial on how to use BRAT (Brat Rapid Annotation Tool), an\n",
    "online environment for collaborative text annotation. \n",
    "\n",
    "http://brat.nlplab.org/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Define a `Candidate` Type\n",
    "\n",
    "We repeat our definition of the `Spouse` `Candidate` subclass from Parts II and III."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass, Document, Candidate\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Select an example `Candidate` and `Document` \n",
    "\n",
    "Candidates are divided into 3 splits, each mapped to a unique integer id:  \n",
    "- 0: _training_  \n",
    "- 1: _development_  \n",
    "- 2: _testing_   \n",
    "\n",
    "In this tutorial, we'll load our training set candidates and create gold labels for a document using the BRAT interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Launching BRAT\n",
    "BRAT runs as as seperate server application. Snorkel will automatically download and configure a BRAT instance for you. When you first initialize this server, you need to provide your applications `Candidate` type. For this tutorial, we use the `Spouse` relation defined above, which consists of a pair of `PERSON` named entities connected by marriage. \n",
    "\n",
    "Currently, we only support 1 relation type per-application. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading BRAT [http://weaver.nlplab.org/~brat/releases/brat-v1.3_Crunchy_Frog.tar.gz]...\n",
      "Installing BRAT...\n",
      "Launching BRAT server at http://localhost:8001 [pid=44647]...\n"
     ]
    }
   ],
   "source": [
    "from snorkel.contrib.brat import BratAnnotator\n",
    "\n",
    "brat = BratAnnotator(session, Spouse, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Initialize our document collection\n",
    "\n",
    "BRAT creates a local copy of all the documents and annotations found in a `split` set. We initialize a document collection by defining a unique set name, _spouse/train_, and then passing in our training set candidates via the `split` id. Annotations are stored as plain text files in [standoff](http://brat.nlplab.org/standoff.html) format.\n",
    "\n",
    "<img align=\"left\" src=\"imgs/brat-login.jpg\" width=\"200px\" style=\"margin-right:50px\">\n",
    "\n",
    "After launching the BRAT annotator for the first time, you will need to login to begin editing annotations. Navigate your mouse to the upper right-hand corner of the BRAT interface (see Fig. 1) click 'login' and enter the following information:\n",
    "\n",
    "- **login**: _brat_\n",
    "- **password**: _brat_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Advanced BRAT users can setup multiple annotator accounts by adding USER/PASSWORD key pairs to the `USER_PASSWORD` dictionary found in `snokel/contrib/brat/brat-v1.3_Crunchy_Frog/config.py`. This is useful if you would like to keep track of multiple annotator judgements for later adjudication or use as labeling functions as per our tutorial on using [Snorkel for Crowdsourcing](https://github.com/HazyResearch/snorkel/blob/master/tutorials/crowdsourcing/Crowdsourced_Sentiment_Analysis.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "brat.init_collection(\"spouse/train\", split=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've already generated some BRAT annotations for you, so let's import an existing collection for purposes of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Imported archive to /Users/fries/code/workshop/snorkel/snorkel/contrib/brat/brat-v1.3_Crunchy_Frog/data/\n"
     ]
    }
   ],
   "source": [
    "brat.import_collection(\"data/brat_spouse.zip\", overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Launch BRAT Interface in a New Window\n",
    "Once our collection is initialized, we can view specific documents for annotation. The default mode is to generate a HTML link to a new BRAT browser window. Click this link to connect to launch the annotator editor. \n",
    "\n",
    "Optionally, you can launch BRAT in an embedded window by calling:\n",
    "\n",
    "    brat.view(\"spouse/train\", doc, new_window=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='http://localhost:8001/index.xhtml#/spouse/train/5ede8912-59c9-4ba9-93df-c58cebb542b7' target='_blank'>Launch BRAT</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc_name = '5ede8912-59c9-4ba9-93df-c58cebb542b7'\n",
    "doc = session.query(Document).filter(Document.name==doc_name).one()\n",
    "\n",
    "brat.view(\"spouse/train\", doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not have a specific document to edit, you can optionally launch BRAT and use their file browser to navigate through all files found in the target collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='http://localhost:8001/index.xhtml#/spouse/train/' target='_blank'>Launch BRAT</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "brat.view(\"spouse/train\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Creating Gold Label Annotations\n",
    "### a) Annotating Named Entities\n",
    "`Spouse` relations consist of 2 `PERSON` named entities. When annotating our validation documents, \n",
    "the first task is to identify our target entities. In this tutorial, we will annotate all `PERSON` \n",
    "mentions found in our example document, though for your application you may choose to only label \n",
    "those that particpate in a true relation. \n",
    "\n",
    "<img align=\"right\" src=\"imgs/brat-anno-dialog.jpg\" width=\"400px\" style=\"margin-left:50px\">\n",
    "\n",
    "Begin by selecting and highlighting the text corresponding to a `PERSON` entity. Once highlighted, an annotation dialog will appear on your screen (see image of the BRAT Annotation Dialog Window to the right). If this is correct, click ok. Repeat this for every entity you find in the document.\n",
    "\n",
    "**Annotation Guidelines**\n",
    "\n",
    "When developing gold label annotations, you should always discuss and agree on a set of _annotator guidelines_ to share with human labelers. These are the guidelines we used to label the `Spouse` relation:\n",
    "\n",
    "- **<span style=\"color:red\">Do not</span>** include formal titles associated with professional roles e.g., _**Pastor** Jeff_, _**Prime Minister** Prayut Chan-O-Cha_\n",
    "- Do include English honorifics unrelated to a professional role, e.g., _**Mr.** John Cleese_.\n",
    "- **<span style=\"color:red\">Do not</span>** include family names/surnames that do not reference a single individual, e.g., _the Duggar family_.\n",
    "- Do include informal titles, stage names, fictional characters, and nicknames, e.g., _**Dog the Bounty Hunter**_\n",
    "- Include possessive's, e.g., _Anna**'s**_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Annotating Relations\n",
    "\n",
    "To annotate `Spouse` relations, we look through all pairs of `PERSON` entities found within a single sentence. BRAT identifies the bounds of each sentence and renders a numbered row in the annotation window (see the left-most column in the image below).  \n",
    "\n",
    "<img align=\"right\" src=\"imgs/brat-relation.jpg\" width=\"500px\" style=\"margin-left:50px\">\n",
    "\n",
    "Annotating relations is done through simple drag and drop. Begin by clicking and holding on a single `PERSON` entity and then drag that entity to its corresponding spouse entity. That is it!\n",
    "\n",
    "**Annotation Guidelines**\n",
    "\n",
    "- Restrict `PERSON` pairs to those found in the same sentence.\n",
    "- The order of `PERSON` arguments does not matter in this application.\n",
    "- **<span style=\"color:red\">Do not</span>** include relations where a `PERSON` argument is wrong or otherwise incomplete."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Scoring Models using BRAT Labels\n",
    "\n",
    "### a) Evaluating System Recall\n",
    "\n",
    "Creating gold validation data with BRAT is a critical evaluation step because it allows us to compute an estimate of our model's _true recall_. When we create labeled data over a candidate set created by Snorkel, we miss mentions of relations that our candidate extraction step misses. This causes us to overestimate the system's true recall.\n",
    "\n",
    "In the code below, we show how to map BRAT annotations to an existing set of Snorkel candidates and compute some associated metrics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_cands = session.query(Spouse).filter(Spouse.split == 0).order_by(Spouse.id).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Mapping BRAT Annotations to Snorkel Candidates\n",
    "We annotated a single document using BRAT to illustrate the difference in scores when we factor in the effects of candidate generation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Mapped 7/14 (50%) of BRAT labels to candidates\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 9s, sys: 1.83 s, total: 1min 11s\n",
      "Wall time: 1min 11s\n"
     ]
    }
   ],
   "source": [
    "%time brat.import_gold_labels(session, \"spouse/train\", train_cands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our candidate extractor only captures 7/14 (50%) of true mentions in this document. Our real system's recall is likely even worse, since we won't correctly predict the label for all true candidates. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c) Re-loading the Trained LSTM\n",
    "We'll load the LSTM model we trained in [Workshop_4_Discriminative_Model_Training.ipynb](Workshop_4_Discriminative_Model_Training.ipynb) and use to to predict marginals for our test candidates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_cands = session.query(Spouse).filter(Spouse.split == 2).order_by(Spouse.id).all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/spouse.lstm/spouse.lstm-0\n",
      "[reRNN] Loaded model <spouse.lstm>\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning.disc_models.rnn import reRNN\n",
    "\n",
    "lstm = reRNN(seed=1701, n_threads=None)\n",
    "lstm.load(\"spouse.lstm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "marginals = lstm.marginals(test_cands)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### d) Create a Subset of Test for Evaluation\n",
    "\n",
    "Our measures assume BRAT annotations are complete for the given set of documents! Rather than manually annotating the entire test set, we define a small subset of 10 test documents for hand lableing.  We'll then compute the full, recall-corrected metrics for this subset.\n",
    "\n",
    "First, let's build a query to initalize this candidate collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error! Collection at 'spouse/test-subset' already exists. Please set overwrite=True to erase all existing annotations.\n"
     ]
    }
   ],
   "source": [
    "doc_ids = set(open(\"data/brat_test_docs.tsv\",\"rb\").read().splitlines())\n",
    "cid_query = [c.id for c in test_cands if c.get_parent().document.name in doc_ids]\n",
    "\n",
    "brat.init_collection(\"spouse/test-subset\", cid_query=cid_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href='http://localhost:8001/index.xhtml#/spouse/test-subset/' target='_blank'>Launch BRAT</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "brat.view(\"spouse/test-subset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### e) Comparing Unadjusted vs. Adjusted Scores "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD8CAYAAACRkhiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD5tJREFUeJzt3X+s3XV9x/Hna+AIU8nqemWslLVkNVkhiqNriJoFwzYQ\n/yhkxtUtQjJC3WBGE/8Y+IeaLE1YMnUhGSyoBEicpIkizQAXZG7EOMQL6SgtdnYC0qZC1WWoWdha\n3vvjfpCza8s9997zg9vP85F8c77n/f1x3h8O7et8f5zTVBWSpD79wrQbkCRNjyEgSR0zBCSpY4aA\nJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6tjJ025gIatXr65169ZNuw1JWlEeeeSRH1TVzELrvepD\nYN26dczOzk67DUlaUZI8Pcx6ng6SpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQ\nkKSOveq/MbwSrbvunmVt/9QN7x5RJ5L0yjwSkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSp\nY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4tGAJJ1ib5WpK9\nSfYk+VCrfyLJwSS72nTpwDbXJ9mfZF+Siwfq5yfZ3ZbdmCTjGZYkaRjD/POSR4CPVNWjSV4PPJLk\n/rbs01X114MrJ9kIbAXOAX4N+GqSN1XVUeBm4Grgm8C9wCXAfaMZiiRpsRY8EqiqQ1X1aJv/MfAE\nsOYVNtkC3FlVL1TVk8B+YHOSM4DTquqhqirgDuCyZY9AkrRki7omkGQd8FbmPskDfDDJY0luTbKq\n1dYAzwxsdqDV1rT5+XVJ0pQMHQJJXgd8EfhwVT3P3Kmds4HzgEPAJ0fVVJJtSWaTzB4+fHhUu5Uk\nzTNUCCR5DXMB8Pmq+hJAVT1bVUer6kXgM8DmtvpBYO3A5me22sE2P7/+c6rqlqraVFWbZmZmFjMe\nSdIiDHN3UIDPAU9U1acG6mcMrHY58Hib3wlsTXJKkvXABuDhqjoEPJ/kgrbPK4C7RzQOSdISDHN3\n0NuB9wO7k+xqtY8C70tyHlDAU8AHAKpqT5IdwF7m7iy6tt0ZBHANcBtwKnN3BXlnkCRN0YIhUFVf\nB451P/+9r7DNdmD7MeqzwLmLaVCSND5+Y1iSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4Z\nApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEg\nSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdWzAEkqxN8rUke5PsSfKhVn9D\nkvuTfKc9rhrY5vok+5PsS3LxQP38JLvbshuTZDzDkiQNY5gjgSPAR6pqI3ABcG2SjcB1wANVtQF4\noD2nLdsKnANcAtyU5KS2r5uBq4ENbbpkhGORJC3SgiFQVYeq6tE2/2PgCWANsAW4va12O3BZm98C\n3FlVL1TVk8B+YHOSM4DTquqhqirgjoFtJElTsKhrAknWAW8FvgmcXlWH2qLvA6e3+TXAMwObHWi1\nNW1+fl2SNCVDh0CS1wFfBD5cVc8PLmuf7GtUTSXZlmQ2yezhw4dHtVtJ0jxDhUCS1zAXAJ+vqi+1\n8rPtFA/t8blWPwisHdj8zFY72Obn139OVd1SVZuqatPMzMywY5EkLdIwdwcF+BzwRFV9amDRTuDK\nNn8lcPdAfWuSU5KsZ+4C8MPt1NHzSS5o+7xiYBtJ0hScPMQ6bwfeD+xOsqvVPgrcAOxIchXwNPBe\ngKrak2QHsJe5O4uuraqjbbtrgNuAU4H72iRJmpIFQ6Cqvg4c737+i46zzXZg+zHqs8C5i2lQkjQ+\nfmNYkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0z\nBCSpY4aAJHXMEJCkjhkCktQxQ0CSOmYISFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNA\nkjpmCEhSxwwBSeqYISBJHVswBJLcmuS5JI8P1D6R5GCSXW26dGDZ9Un2J9mX5OKB+vlJdrdlNybJ\n6IcjSVqMYY4EbgMuOUb901V1XpvuBUiyEdgKnNO2uSnJSW39m4GrgQ1tOtY+JUkTtGAIVNWDwI+G\n3N8W4M6qeqGqngT2A5uTnAGcVlUPVVUBdwCXLbVpSdJoLOeawAeTPNZOF61qtTXAMwPrHGi1NW1+\nfl2SNEVLDYGbgbOB84BDwCdH1hGQZFuS2SSzhw8fHuWuJUkDlhQCVfVsVR2tqheBzwCb26KDwNqB\nVc9stYNtfn79ePu/pao2VdWmmZmZpbQoSRrCkkKgneN/yeXAS3cO7QS2JjklyXrmLgA/XFWHgOeT\nXNDuCroCuHsZfUuSRuDkhVZI8gXgQmB1kgPAx4ELk5wHFPAU8AGAqtqTZAewFzgCXFtVR9uurmHu\nTqNTgfvaJEmaogVDoKred4zy515h/e3A9mPUZ4FzF9WdJGms/MawJHXMEJCkjhkCktQxQ0CSOmYI\nSFLHDAFJ6pghIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAk\ndcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOrZgCCS5\nNclzSR4fqL0hyf1JvtMeVw0suz7J/iT7klw8UD8/ye627MYkGf1wJEmLMcyRwG3AJfNq1wEPVNUG\n4IH2nCQbga3AOW2bm5Kc1La5Gbga2NCm+fuUJE3YgiFQVQ8CP5pX3gLc3uZvBy4bqN9ZVS9U1ZPA\nfmBzkjOA06rqoaoq4I6BbSRJU7LUawKnV9WhNv994PQ2vwZ4ZmC9A622ps3Pr0uSpmjZF4bbJ/sa\nQS8/k2Rbktkks4cPHx7lriVJA5YaAs+2Uzy0x+da/SCwdmC9M1vtYJufXz+mqrqlqjZV1aaZmZkl\ntihJWshSQ2AncGWbvxK4e6C+NckpSdYzdwH44Xbq6PkkF7S7gq4Y2EaSNCUnL7RCki8AFwKrkxwA\nPg7cAOxIchXwNPBegKrak2QHsBc4AlxbVUfbrq5h7k6jU4H72iRJmqIFQ6Cq3necRRcdZ/3twPZj\n1GeBcxfVnSRprPzGsCR1zBCQpI4ZApLUMUNAkjpmCEhSxxa8O0iTt+66e5a87VM3vHuEnUg60Xkk\nIEkdMwQkqWOGgCR1zBCQpI4ZApLUMUNAkjpmCEhSxwwBSeqYISBJHTMEJKljhoAkdcwQkKSOGQKS\n1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkCktQxQ0CSOrasEEjyVJLdSXYlmW21\nNyS5P8l32uOqgfWvT7I/yb4kFy+3eUnS8oziSOCdVXVeVW1qz68DHqiqDcAD7TlJNgJbgXOAS4Cb\nkpw0gteXJC3ROE4HbQFub/O3A5cN1O+sqheq6klgP7B5DK8vSRrSckOggK8meSTJtlY7vaoOtfnv\nA6e3+TXAMwPbHmg1SdKUnLzM7d9RVQeTvBG4P8m3BxdWVSWpxe60Bco2gLPOOmuZLUqSjmdZRwJV\ndbA9PgfcxdzpnWeTnAHQHp9rqx8E1g5sfmarHWu/t1TVpqraNDMzs5wWJUmvYMkhkOS1SV7/0jzw\n+8DjwE7gyrbalcDdbX4nsDXJKUnWAxuAh5f6+pKk5VvO6aDTgbuSvLSfv6+qryT5FrAjyVXA08B7\nAapqT5IdwF7gCHBtVR1dVveSpGVZcghU1XeBtxyj/kPgouNssx3YvtTXlCSN1nIvDJ+w1l13z7Rb\nkKSx82cjJKljhoAkdcwQkKSOGQKS1DFDQJI6ZghIUscMAUnqmCEgSR0zBCSpY4aAJHXMEJCkjhkC\nktQxQ0CSOmYISFLHDAFJ6pghIEkd8x+VOcEs5x/DeeqGd4+wE0krgUcCktQxQ0CSOubpIEl6BSf6\nKVaPBCSpYx4JSHrVW86ncVgZn8inxSMBSeqYRwKSJmK5n+ZX4muvhOsJhoCkoU3zL3KNhyEgrUD+\nZaxRMQSkKfAvcb1aTPzCcJJLkuxLsj/JdZN+fUnSyyZ6JJDkJOBvgd8DDgDfSrKzqvZOsg/pJSvh\nwp00TpM+HbQZ2F9V3wVIciewBRhLCHjIvTj+91oc/3vpRDDp00FrgGcGnh9oNUnSFLwqLwwn2QZs\na09/kmTfNPsZsdXAD6bdxIQ55hNfb+OFMY85f7XsXfz6MCtNOgQOAmsHnp/Zav9PVd0C3DKppiYp\nyWxVbZp2H5PkmE98vY0XTpwxT/p00LeADUnWJ/lFYCuwc8I9SJKaiR4JVNWRJH8O/CNwEnBrVe2Z\nZA+SpJdN/JpAVd0L3Dvp130VOSFPcy3AMZ/4ehsvnCBjTlVNuwdJ0pT4U9KS1DFDYEwW+nmMJH+c\n5LEku5N8I8lbptHnKA0x5i1tzLuSzCZ5xzT6HJVhfwIlyW8nOZLkPZPsbxyGeI8vTPJf7T3eleRj\n0+hzlIZ5n9u4dyXZk+RfJt3jslSV04gn5i56/wdwNvCLwL8BG+et8zZgVZt/F/DNafc9gTG/jpdP\nQb4Z+Pa0+x7neAfW+yfmroO9Z9p9T+A9vhD4h2n3OuEx/zJzv3pwVnv+xmn3vZjJI4Hx+NnPY1TV\n/wAv/TzGz1TVN6rqP9vTh5j7zsRKNsyYf1LtTwnwWmAlX5BacLzNB4EvAs9NsrkxGXbMJ5JhxvxH\nwJeq6nsAVbWi3mtDYDwW+/MYVwH3jbWj8RtqzEkuT/Jt4B7gTybU2zgsON4ka4DLgZsn2Nc4Dfv/\n9dvaab/7kpwzmdbGZpgxvwlYleSfkzyS5IqJdTcCr8qfjehJkncyFwIr+vz4sKrqLuCuJL8D/CXw\nu1NuaZz+BviLqnoxybR7mZRHmTst8pMklwJfBjZMuadxOxk4H7gIOBX41yQPVdW/T7et4RgC4zHU\nz2MkeTPwWeBdVfXDCfU2LkON+SVV9WCSs5OsrqqV+Jszw4x3E3BnC4DVwKVJjlTVlyfT4sgtOOaq\nen5g/t4kN63g9xiGe58PAD+sqp8CP03yIPAWYEWEwNQvSpyIE3Ph+l1gPS9fTDpn3jpnAfuBt027\n3wmO+Td4+cLwbzH3hynT7n1c4523/m2s/AvDw7zHvzrwHm8GvrdS3+NFjPk3gQfaur8EPA6cO+3e\nh508EhiDOs7PYyT507b874CPAb8C3NQ+KR6pFfxjVEOO+Q+AK5L8L/DfwB9W+1O00gw53hPKkGN+\nD/BnSY4w9x5vXanvMQw35qp6IslXgMeAF4HPVtXj0+t6cfzGsCR1zLuDJKljhoAkdcwQkKSOGQKS\n1DFDQJI6ZghIUscMAUnqmCEgSR37P6/rPa5f2cLdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11e456650>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.hist(marginals, bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev  = load_gold_labels(session, annotator_name='gold', split=1, load_as_array=True, zero_one=True)\n",
    "L_gold_test = load_gold_labels(session, annotator_name='gold', split=2, zero_one=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Recall-uncorrected Score** If we don't account for candidates missed during extraction, our model score will overestimate real performance, as is the case for the model evaluation below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Unadjusted BRAT Scores (10 Documents)\n",
      "========================================\n",
      "Pos. class accuracy: 0.75\n",
      "Neg. class accuracy: 0.852\n",
      "Precision            0.25\n",
      "Recall               0.75\n",
      "F1                   0.375\n",
      "----------------------------------------\n",
      "TP: 6 | FP: 18 | TN: 104 | FN: 2\n",
      "========================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Mapped 8/11 (73%) of BRAT labels to candidates\n"
     ]
    }
   ],
   "source": [
    "brat.score(session, test_cands, marginals, \"spouse/test-subset\", recall_correction=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Recall-corrected Score** Though this is a small sample of documents, we see how missing candidates can impact our real system score. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Adjusted BRAT Scores (10 Documents)\n",
      "========================================\n",
      "Pos. class accuracy: 0.545\n",
      "Neg. class accuracy: 0.852\n",
      "Precision            0.25\n",
      "Recall               0.545\n",
      "F1                   0.343\n",
      "----------------------------------------\n",
      "TP: 6 | FP: 18 | TN: 104 | FN: 5\n",
      "========================================\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Mapped 8/11 (73%) of BRAT labels to candidates\n"
     ]
    }
   ],
   "source": [
    "brat.score(session, test_cands, marginals, \"spouse/test-subset\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the full model, evaluated on all our gold candidate labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 0.627\n",
      "Neg. class accuracy: 0.923\n",
      "Precision            0.426\n",
      "Recall               0.627\n",
      "F1                   0.507\n",
      "----------------------------------------\n",
      "TP: 281 | FP: 379 | TN: 4567 | FN: 167\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = lstm.error_analysis(session, test_cands, L_gold_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}