tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scholar is a framework for document modeling, much like LDA, but with the ability to flexibly incorporate metadata, with some similarity to the structural topic model. It can scale to large numbers of covariates, runs in python, and offers GPU support for fast exploration of a corpus of documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll use a toy corpus of political press releases in order to demonstrate the functionality and interface of Scholar."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code\n",
    "\n",
    "If you're looking at this tutorial, you have presumably already downloaded/cloned the Scholar repo. If not, you can clone it using: \n",
    "\n",
    "`git clone git@github.com:dallascard/scholar.git`\n",
    "\n",
    "(if you don't have git set up, you can just download the repo from https://github.com/dallascard/scholar)\n",
    "\n",
    "Scholar has not yet been packaged us as a full python packge. As such, we will just run commands from the scholar directory, so switch into it:\n",
    "\n",
    "`cd /path/to/scholar`\n",
    "\n",
    "(the directory that contains this notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment\n",
    "\n",
    "Create new python environment for running scholar. We recommend using Anaconda, but it is also possible to do this using virtualenv and pip. Assuming you are using conda, run the following three commands, one at a time, and follow the prompts:\n",
    "\n",
    "`conda create -n scholar python=3`\n",
    "\n",
    "`source activate scholar`\n",
    "\n",
    "`conda install ipython notebook numpy=1.15.4 scipy=1.1.0 pandas matplotlib gensim pytorch=1.0.0 torchvision -c pytorch`\n",
    "\n",
    "You should also now quit this notebook and reload it from with the scholar environment (i.e. after running `source activate scholar` in the shell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "For this tutorial, we'll be using a subset of the Congressional press release corpus created by Justin Grimmer.\n",
    "\n",
    "A compressed file with press releases from six senators can be found in this repo. To expand it, run\n",
    "\n",
    "`tar -xzf tutorial.tar.gz`\n",
    "\n",
    "which will create a  directory called  `tutorial/CongressPressExpand/`\n",
    "\n",
    "For those who are interested, the full dataset (Press.tar; 282Mb) can be downloaded from https://dataverse.harvard.edu/dataset.xhtml?persistentId=hdl:1902.1/14596"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note about reproducibility\n",
    "\n",
    "The results in this notebook should be basically reproducible, however, it seems that this depends on having the same version of certain packages. The code will still work with different versions, but the resulting topics may be different. Moreover, the results will be slightly different if using a GPU, and there may also be very small numerical differences on different machines.\n",
    "\n",
    "The results given here were obtained using the following environment:\n",
    "- python 3.7.1\n",
    "- pytorch 1.0.0\n",
    "- numpy 1.15.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Import\n",
    "\n",
    "To use scholar, it is necessary to get the data into the proper format. To do this, the easiest thing to do is to write a script to convert the documents into a single file, where each line corresponds to one document, represented as a JSON object. Each JSON object should have at least one field called \"text\", but it can also contain other metadata fields.\n",
    "\n",
    "We have included a script to convert a subset of the Senatorial press releases into this format, which can be used as a starting point for your own project.\n",
    "\n",
    "If running this from the command line, it would be run as\n",
    "\n",
    "`python import_congress_press.py tutorial/CongressPressExpand tutorial/congress/`\n",
    "\n",
    "Since we are running this in a notebook, we will run it by importing the package, and calling the `main()` function with the corresponding arguments in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python import_congress_press.py tutorial/CongressPressExpand tutorial/congress\n",
      "274 files from Sanders\n",
      "709 files from Obama\n",
      "358 files from Klobuchar\n",
      "293 files from McCain\n",
      "614 files from Graham\n",
      "235 files from Coburn\n",
      "Saving files to tutorial/congress\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "import import_congress_press\n",
    "script = 'import_congress_press.py'\n",
    "args = 'tutorial/CongressPressExpand tutorial/congress'\n",
    "print(\"python\", script, args)\n",
    "import_congress_press.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates two files in the output directory (`/tutorial/congress/`):\n",
    "- `train.jsonlist` contains one press release per line, in JSON format, including fields for the text of the press release, as well as senator name, party, date, year, and month\n",
    "- `train.score.csv` contains DW-nominate scores for 6 senators from the 110th congress, with one score for each document in `train.jsonlist` (in the same order)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To inspect a document, we can use the `json` library to load it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id : 10Apr2007Sanders238.txt\n",
      "text : SEIZE THE OPPORTUNITY FOR MAJOR BREAKTHROUGHS IN HEALTH CARE   The Senate this week is considering a bill, cosponsored by Sen. Bernie Sanders, that would expand and encourage federal funding of human embryonic stem cell research. President Bush in 2001 cut off federal funding for research involving new embryonic stem cells, which has dramatically stalled this critical area of medical research.      \"My hope is that, as a result of increased pressure from scientists, physicians and the American people, the president will change his position or, if he does not, that the Congress will have enough votes to override his veto and establish unrestricted federal funding for stem cell research,\" Sanders said. \"The potential is now available for major breakthroughs in Parkinson's disease, Alzheimer's, diabetes, spinal cord injury, stroke, heart disease and many other illnesses. We must seize the opportunity.\"     View a copy of the Bill - S. 5 at http://sanders.senate.gov/files/S_5.pdf    Read a summary of the bill - S. 5 at http://www.sanders.senate.gov/news/record.cfm?id=272151    Read a Congressional Research Service background report on stem cell research at http://sanders.senate.gov/files/Stem%20Cells.pdf.    \n",
      "senator : Sanders\n",
      "date : 10Apr2007\n",
      "year : 2007\n",
      "month : Apr\n",
      "party : D\n",
      "score : -0.717\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "with open(os.path.join('tutorial', 'congress', 'train.jsonlist')) as f:\n",
    "    lines = f.readlines()\n",
    "first_doc = json.loads(lines[0])\n",
    "for key, value in first_doc.items():\n",
    "    print(key, ':', value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had documents that we wanted to use as a test set, we could create a `test.jsonlist` object in the same manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to use a preprocessing script we have provided to convert the documents from JSON format into a numerical representation. It works by creating a vocabulary, filtering out numbers, punctuation, and some other tokens, and saving the document-term count matrix. \n",
    "\n",
    "The preprocessing script can also simultaneously pull out various metadata attributes (like author or year) from the JSON objects, and convert them into one-hot representations, which will be saved as .csv files.\n",
    "\n",
    "For other types of metadata, such as continously-valued data, you will need to create the corresponding .csv files manually, as we did above for `train.score.csv`. All that matters is that the order of the rows is the same as the the order of documents in `train.jsonlist`. Also, scholar will expect the file name to be `train.field_name.csv` (or \"test\" rather than \"train\" for test data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see the full set of options for any command, try running it with the `-h` option. Again, in the shell, this would be run as:\n",
    "\n",
    "`python preprocess_data.py -h`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python preprocess_data.py -h\n",
      "Usage: ipykernel_launcher.py train.jsonlist output_dir\n",
      "\n",
      "Options:\n",
      "  -h, --help            show this help message and exit\n",
      "  --label=LABEL         field(s) to use as label (comma-separated):\n",
      "                        default=none\n",
      "  --test=TEST           Test data (test.jsonlist): default=none\n",
      "  --train-prefix=TRAIN_PREFIX\n",
      "                        Output prefix for training data: default=train\n",
      "  --test-prefix=TEST_PREFIX\n",
      "                        Output prefix for test data: default=test\n",
      "  --stopwords=STOPWORDS\n",
      "                        List of stopwords to exclude [None|mallet|snowball]:\n",
      "                        default=snowball\n",
      "  --min-doc-count=MIN_DOC_COUNT\n",
      "                        Exclude words that occur in less than this number of\n",
      "                        documents\n",
      "  --max-doc-freq=MAX_DOC_FREQ\n",
      "                        Exclude words that occur in more than this proportion\n",
      "                        of documents\n",
      "  --keep-num            Keep tokens made of only numbers: default=False\n",
      "  --keep-alphanum       Keep tokens made of a mixture of letters and numbers:\n",
      "                        default=False\n",
      "  --strip-html          Strip HTML tags: default=False\n",
      "  --no-lower            Do not lowercase text: default=False\n",
      "  --min-length=MIN_LENGTH\n",
      "                        Minimum token length: default=3\n",
      "  --vocab-size=VOCAB_SIZE\n",
      "                        Size of the vocabulary (by most common, following\n",
      "                        above exclusions): default=none\n",
      "  --seed=SEED           Random integer seed (only relevant for choosing test\n",
      "                        set): default=42\n"
     ]
    },
    {
     "ename": "SystemExit",
     "evalue": "0",
     "output_type": "error",
     "traceback": [
      "An exception has occurred, use %tb to see the full traceback.\n",
      "\u001b[0;31mSystemExit\u001b[0m\u001b[0;31m:\u001b[0m 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dcard/anaconda/envs/scholar/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3275: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.\n",
      "  warn(\"To exit: use 'exit', 'quit', or Ctrl-D.\", stacklevel=1)\n"
     ]
    }
   ],
   "source": [
    "import preprocess_data\n",
    "args = '-h'\n",
    "print(\"python preprocess_data.py -h\")\n",
    "preprocess_data.main([args])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we'll tell it to only use words that occur in at least 90 documents, which will give us a vocabulary of about 1000 words (excluding stopwords). This will help the model run faster, but using a larger vocabulary (2000 or 5000 words, might lead to much richer topics). Alternatively, we could set the vocabulary size directly, using `--vocab-size`\n",
    "\n",
    "We'll also tell it to create the label matrices for the various metadata attributes, which we provide in a comma-separated list (without spaces). As a reminder, it will assume that each of these metadata names will be a field in each JSON document object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python preprocess_data.py tutorial/congress/train.jsonlist tutorial/congress --min-doc-count 90 --label senator,party,year,month,date\n",
      "Using snowball stopwords\n",
      "Reading data files\n",
      "Found 2483 training documents\n",
      "Found label senator with 6 classes\n",
      "Found label party with 2 classes\n",
      "Found label year with 4 classes\n",
      "Found label month with 12 classes\n",
      "Found label date with 918 classes\n",
      "Parsing 2483 documents\n",
      "Size of full vocabulary=22131\n",
      "Selecting the vocabulary\n",
      "Vocab size after filtering = 1021\n",
      "Final vocab size = 1021\n",
      "Most common words remaining: senator washington today said senate contact press also current date\n",
      "Converting to count representations\n",
      "Size of train document-term matrix: (2483, 1021)\n",
      "0 words missing from training data\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "script = 'preprocess_data.py'\n",
    "args = 'tutorial/congress/train.jsonlist tutorial/congress --min-doc-count 90 --label senator,party,year,month,date'\n",
    "print(\"python\", script, args)\n",
    "preprocess_data.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This preprocessing script creates several files, some uesd by Scholar, and some designed for other software, like Mallet. The files of interest are:\n",
    "- `train.npz`, which contains a (D x V) sparse matrix of document word counts,\n",
    "- `train.vocab.json`, which contains the vocabualry as a JSON object, and;\n",
    "- files like `train.year.csv`, which contain the year corresponding to each document, in a matrix of size (D x C), where C is the number of distinct covariate values (e.g. years)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a couple of files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First few words in the vocbulary:\n",
      "['ability', 'able', 'abuse', 'access', 'according', 'accountability', '...']\n"
     ]
    }
   ],
   "source": [
    "# load the vocabualry\n",
    "with open(os.path.join('tutorial', 'congress', 'train.vocab.json')) as f:\n",
    "    vocab = json.load(f)\n",
    "print(\"First few words in the vocbulary:\")\n",
    "print(vocab[:6] + ['...'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start of a covariate file (train.year.csv):\n",
      "                         2005  2006  2007  2008\n",
      "10Apr2007Sanders238.txt     0     0     1     0\n",
      "10Apr2008Sanders2.txt       0     0     0     1\n",
      "10Apr2008Sanders3.txt       0     0     0     1\n",
      "10Dec2007Sanders61.txt      0     0     1     0\n",
      "10May2007Sanders212.txt     0     0     1     0\n"
     ]
    }
   ],
   "source": [
    "# load a covariate file\n",
    "import pandas as pd\n",
    "print(\"Start of a covariate file (train.year.csv):\")\n",
    "df = pd.read_csv(os.path.join('tutorial', 'congress', 'train.year.csv'), header=0, index_col=0)\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if we had a test corpus, we could simultaneously process it by adding the `--test` option to our call to `preprocess_data` (with the path to `test.jsonlist`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Scholar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to run Scholar!\n",
    "\n",
    "To start off, let's just run a basic topic model, without any metadata. We just need to specify the input directory,  and it will look for the `train.npz` file, as well as `train.vocab.json`. \n",
    "\n",
    "Here, we'll also tell it to use 10 topics (`-k`), tell it only to run for 50 epochs (`--epochs 50`), and to use a random 1/10th of the data as a dev/validation set to monitor the fit (`--dev-folds 10`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 10 --epochs 50 --dev-folds 10 --seed 42\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 90 5080\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 10\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 0\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 0\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: False\n",
      "Optimizing full model\n",
      "Epoch: 10 cost= 1132.716733519\n",
      "Epoch: 10; Dev perplexity = 983.5218\n",
      "Epoch: 20 cost= 1128.470428945\n",
      "Epoch: 20; Dev perplexity = 857.5101\n",
      "Epoch: 30 cost= 1117.978690401\n",
      "Epoch: 30; Dev perplexity = 798.9625\n",
      "Epoch: 40 cost= 1090.164661109\n",
      "Epoch: 40; Dev perplexity = 756.3840\n",
      "Background frequencies of top words:\n",
      "obama senator bill said senate today graham washington press president\n",
      "[0.01298087 0.01290677 0.00907639 0.00863688 0.00794695 0.00677663\n",
      " 0.00639589 0.00639589 0.00609181 0.00601004]\n",
      "Topics:\n",
      "0: barack contact statement release petitions initiated immediate newsletters  / provide year national minnesota program said bill million ; sparsity=0.0000\n",
      "1: iraq president american war time one iraqi troops  / million announced announces schools funds program announce county ; sparsity=0.0000\n",
      "2: washington mccain browse statement record contact following john  / year program provide national bill help funding one ; sparsity=0.0000\n",
      "3: press graham awarded releases grants fire browse south  / committee government act report america barack many need ; sparsity=0.0000\n",
      "4: energy renewable klobuchar fuel said fuels gas oil  / relases lindsey hickman releases kevin bishop press browse ; sparsity=0.0000\n",
      "5: veterans care health obama members service legislation mental  / record releases wes browse kevin lindsey relases bishop ; sparsity=0.0000\n",
      "6: bill funding million klobuchar year senate said project  / kevin wes relases lindsey bishop date hickman releases ; sparsity=0.0000\n",
      "7: press releases browse carolina kevin relases record south  / barack obama government office states issue people immediate ; sparsity=0.0000\n",
      "8: obama barack federal durbin state assistance lead illinois  / browse relases bishop releases press wes lindsey hickman ; sparsity=0.0000\n",
      "9: year percent tax president americans income american congress  / wes browse releases hickman bishop kevin south relases ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Dev perplexity = 724.7339\n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "import run_scholar\n",
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 10 --epochs 50 --dev-folds 10 --seed 42'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To display this a bit more cleanly, let's load the output and look at it.\n",
    "\n",
    "By default, the output of the model is saved to a directory called `output`, but that can be changed using the `-o` option (remember to use `-h` to see all options).\n",
    "\n",
    "First, let's inspect the background frequencies of the most common words (the log-frequencies are computed and saved)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "obama 0.012980873\n",
      "senator 0.012906771\n",
      "bill 0.0090763895\n",
      "said 0.008636881\n",
      "senate 0.007946953\n",
      "today 0.0067766286\n",
      "graham 0.00639589\n",
      "washington 0.00639589\n",
      "press 0.00609181\n",
      "president 0.0060100416\n",
      "legislation 0.005859277\n",
      "health 0.0057085156\n",
      "federal 0.005634414\n",
      "barack 0.005565421\n",
      "million 0.0052051255\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "\n",
    "# load the background log-frequencies\n",
    "bg = np.load('output/bg.npz')['bg']\n",
    "\n",
    "# load the vocabualry\n",
    "with open('output/vocab.json') as f:\n",
    "    vocab = json.load(f)\n",
    "\n",
    "# sort terms by log-frequency\n",
    "order = np.argsort(bg)\n",
    "\n",
    "# print the most common words \n",
    "for i in range(1, 16):\n",
    "    index = order[-i]\n",
    "    print(vocab[index], np.exp(bg[index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems like about what we would expect for the most common words, including common words, like \"said\", corpus-specific words, like \"senator\", and the names of some of the Senators we have included.\n",
    "\n",
    "Typically in topic models, we might need to remove stopwords to get good topics, but in Scholar the background term means that we don't particularly need to worry about it. (Note that the prepocessing script removed some very common words like \"the\", but they could equally have been left in).\n",
    "\n",
    "Now let's look at the topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: barack contact statement release petitions initiated immediate ; sparsity=0.0000\n",
      "1: iraq president american war time one iraqi ; sparsity=0.0000\n",
      "2: washington mccain browse statement record contact following ; sparsity=0.0000\n",
      "3: press graham awarded releases grants fire browse ; sparsity=0.0000\n",
      "4: energy renewable klobuchar fuel said fuels gas ; sparsity=0.0000\n",
      "5: veterans care health obama members service legislation ; sparsity=0.0000\n",
      "6: bill funding million klobuchar year senate said ; sparsity=0.0000\n",
      "7: press releases browse carolina kevin relases record ; sparsity=0.0000\n",
      "8: obama barack federal durbin state assistance lead ; sparsity=0.0000\n",
      "9: year percent tax president americans income american ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "from run_scholar import print_top_words\n",
    "\n",
    "# load the stored (K x V) topic matrix (stored in a compressed numpy format)\n",
    "beta = np.load(os.path.join('output', 'beta.npz'))['beta']\n",
    "print_top_words(beta, vocab, n_pos=7, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is some order here, with certain words going together, like \"bill\" and \"funding\", or \"press\" and \"release\", but overall these topics are not great.\n",
    "\n",
    "One problem is that we are still seeing names like \"Obama\" and \"Barack\" appearing in the topics, which is not quite what we want. To deal with this, let's add topic covariates, to introduce explicit term for each Senator, to collect the words that are more or less common overall for each one.\n",
    "\n",
    "We'll also let the model run for more epochs, to help it converge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding covariates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we introduce the `--topic-covars` option, which introduces topic-like deviations for observed one-hot covariates. Thus, in addition to the 10 topics, we will here get 6 more vectors of word weights, one for each Senator.\n",
    "\n",
    "When we add the option `--topic-covars senator`, it will look for a file called `train.senator.csv`, with one row for each document, in the same order as `train.jsonlist`. (As a reminder, this file was created by `preprocess_data.py`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Loading covariates from tutorial/congress/train.senator.csv\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 90 5080\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 10\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 0\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 6\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: False\n",
      "Optimizing full model\n",
      "Epoch: 10 cost= 1164.963810385\n",
      "Epoch: 10; Dev perplexity = 936.3517\n",
      "Epoch: 20 cost= 1131.093831926\n",
      "Epoch: 20; Dev perplexity = 823.7403\n",
      "Epoch: 30 cost= 1098.462667785\n",
      "Epoch: 30; Dev perplexity = 724.9147\n",
      "Epoch: 40 cost= 1048.331285174\n",
      "Epoch: 40; Dev perplexity = 674.0774\n",
      "Epoch: 50 cost= 1079.597725068\n",
      "Epoch: 50; Dev perplexity = 636.2749\n",
      "Epoch: 60 cost= 1076.581337607\n",
      "Epoch: 60; Dev perplexity = 613.9344\n",
      "Epoch: 70 cost= 1058.620064755\n",
      "Epoch: 70; Dev perplexity = 583.6807\n",
      "Epoch: 80 cost= 1047.979280271\n",
      "Epoch: 80; Dev perplexity = 565.5054\n",
      "Epoch: 90 cost= 1107.038819998\n",
      "Epoch: 90; Dev perplexity = 552.1917\n",
      "Epoch: 100 cost= 1062.782913652\n",
      "Epoch: 100; Dev perplexity = 538.1315\n",
      "Epoch: 110 cost= 1070.185000699\n",
      "Epoch: 110; Dev perplexity = 524.4316\n",
      "Epoch: 120 cost= 1059.096663302\n",
      "Epoch: 120; Dev perplexity = 506.2675\n",
      "Epoch: 130 cost= 1029.882523027\n",
      "Epoch: 130; Dev perplexity = 489.2117\n",
      "Epoch: 140 cost= 1088.313261369\n",
      "Epoch: 140; Dev perplexity = 475.4750\n",
      "Background frequencies of top words:\n",
      "obama senator bill said senate today graham washington press president\n",
      "[0.01298087 0.01290677 0.00907639 0.00863688 0.00794695 0.00677663\n",
      " 0.00639589 0.00639589 0.00609181 0.00601004]\n",
      "Topics:\n",
      "0: statement initiated opinion following newsletters polls petitions pursuant  / increase total based number equipment amount additional available ; sparsity=0.0000\n",
      "1: energy gas oil renewable fuels reduce fire sources  / office afghanistan iraq terror pursuant military primary criminal ; sparsity=0.0000\n",
      "2: consumer safety commerce commission lead children consumers report  / opportunities wes relief community businesses rural economy students ; sparsity=0.0000\n",
      "3: income tax insurance low health americans priorities families  / general releases supreme wes department east meeting justice ; sparsity=0.0000\n",
      "4: troops iraq war course mission iraqi terror success  / federal department health grants program receive facilities purchase ; sparsity=0.0000\n",
      "5: emergency request residents durbin recovery damage federal funds  / ben health american americans thing instead debate nation ; sparsity=0.0000\n",
      "6: nomination supreme judge judiciary justice rights man court  / bill billion appropriations fiscal million projects legislation grants ; sparsity=0.0000\n",
      "7: earmarks immigration transparency judiciary reform let point floor  / health awarded announced grant community receive east purchase ; sparsity=0.0000\n",
      "8: veterans affairs defense afghanistan military members returning soldiers  / interest countries economy wes reach business citizens public ; sparsity=0.0000\n",
      "9: announces county project announced projects awarded development south  / statement united fact issue last instead states even ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Covariate deviations:\n",
      "Coburn: coburn tom added citizen spending released subcommittee room  / wes barack minnesota graham south sanders date thursday ; sparsity=0.0000\n",
      "Graham: wes releases press hickman relases graham date carolina  / barack thursday immediate amy friday added sanders vietor ; sparsity=0.0000\n",
      "Klobuchar: minnesota klobuchar amy commerce renewable businesses incentives farmers  / barack date wes releases vermont press browse pursuant ; sparsity=0.0000\n",
      "McCain: mccain john simply freedom though record authorized browse  / gibbs amy thursday ben klobuchar south vietor wes ; sparsity=0.0000\n",
      "Obama: release immediate barack illinois pursuant julian ortiz alerts  / press klobuchar browse carolina hickman wes south vermont ; sparsity=0.0000\n",
      "Sanders: vermont bernie sanders read http sen told food  / wes klobuchar date amy graham pursuant south obama ; sparsity=0.0000\n",
      "sparsity in covariates = 0.0000\n",
      "Dev perplexity = 466.5493\n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, let's load the topics and have a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: statement initiated opinion following newsletters polls petitions ; sparsity=0.0000\n",
      "1: energy gas oil renewable fuels reduce fire ; sparsity=0.0000\n",
      "2: consumer safety commerce commission lead children consumers ; sparsity=0.0000\n",
      "3: income tax insurance low health americans priorities ; sparsity=0.0000\n",
      "4: troops iraq war course mission iraqi terror ; sparsity=0.0000\n",
      "5: emergency request residents durbin recovery damage federal ; sparsity=0.0000\n",
      "6: nomination supreme judge judiciary justice rights man ; sparsity=0.0000\n",
      "7: earmarks immigration transparency judiciary reform let point ; sparsity=0.0000\n",
      "8: veterans affairs defense afghanistan military members returning ; sparsity=0.0000\n",
      "9: announces county project announced projects awarded development ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "beta = np.load('output/beta.npz')['beta']\n",
    "print_top_words(beta, vocab, n_pos=7, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These look much better! There are fairly clear topics for the energy, Iraq, infrastructure, etc.\n",
    "\n",
    "In addition, we can load the vectors that have been learned for each Senator, which will be saved in a file called `beta_c.npz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coburn: coburn tom added citizen spending released subcommittee ; sparsity=0.0000\n",
      "Graham: wes releases press hickman relases graham date ; sparsity=0.0000\n",
      "Klobuchar: minnesota klobuchar amy commerce renewable businesses incentives ; sparsity=0.0000\n",
      "McCain: mccain john simply freedom though record authorized ; sparsity=0.0000\n",
      "Obama: release immediate barack illinois pursuant julian ortiz ; sparsity=0.0000\n",
      "Sanders: vermont bernie sanders read http sen told ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "topic_covars = np.load(os.path.join('output', 'beta_c.npz'))\n",
    "weights = topic_covars['beta']\n",
    "names = topic_covars['names']\n",
    "print_top_words(weights, vocab, topic_names=names, n_pos=7, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are clearly about right, in that the Senator's names and/or states are appearing in the right vectors. It is hard to make sense of some of these, but they appear to doing the right thing in terms of pulling the Senator-specific terms out of the topics.\n",
    "\n",
    "Note that if we had used the full dataset, we could easily extend the covariates to include a variable for each Senator without difficulty, which would be quite slow to run in the structural topic model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing results\n",
    "\n",
    "Let's load the resulting document-topic proportions for the training data, and look a random example. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10Apr2008Sanders2.txt\n",
      "SENATE ADOPTS SANDERS VETERANS PROVISION IN HOUSING BILL   The Senate today put finishing touches on housing stimulus legislation, adding a $57 million proposal authored by Senator Bernie Sanders (I-Vt.) that would increase federal grants to help disabled veterans adapt their homes.     \"With so many soldiers coming back from Iraq with disabilities, it is absolutely imperative that we make sure they have as normal a life as possible and that certainly includes adapting their homes to meet their needs,\" Sanders said.    The final bill, which the Senate approved 84 to 12, also included energy tax credits cosponsored by Sanders that would promote renewable energy and energy efficiency. It would extend expiring federal tax benefits for investment in solar, wind and other sustainable energy sources.    Sanders also was the lead cosponsor of a successful amendment by Senator Patrick Leahy (D-Vt.) that would guarantee Vermont a $20 million share of $4 billion in community development block grants to prevent home foreclosures and to refurbish abandoned homes.    Under the veterans amendment, veterans with certain severe service-connected disabilities would be eligible for grants of up to $60,000, a $10,000 boost from the current law, to build wheelchair ramps and to make other changes so they could live at home. Veterans who were blinded or lost arms in war zones or while on active duty may receive up to $12,000, a $2,000 increase, for remodeling their homes.    The amendment also would provide for automatic annual adjustments pegged to a home construction cost index to ensure that the benefits keep pace with rising prices. Wounded veterans returning from Iraq and Afghanistan have found that the program now on the books does not cover all of the costs of adapting their homes.    Sanders' amendment was supported by the American Legion, Veterans of Foreign Wars, Disabled American Veterans, Vietnam Veterans of America, AMVETS, Paralyzed Veterans of America and others.    Before passing the bill, the Senate also added the energy provisions. \"There are huge opportunities that we will lose if we do not extend these sustainable energy tax credits,\" Sanders said. \"Not only will these tax credits enable us to continue our effort to break our dependency on the fossil fuels that cause global warming and pollute our environment, but they will provide a significant number of good-paying green jobs in the areas of wind, solar and geothermal - something that I am working very hard on and believe has tremendous potential for our economy.\"    The Leahy-Sanders Amendment would allot $20 million to help Vermont deal with the foreclosure crisis that is sweeping the country. \"With this funding, it is my hope that Vermont's cities and towns will be able to provide immediate assistance to the struggling middle class trying to hold onto their homes and improve communities hit hard by foreclosures,\" Sanders said. \"Clearly, we must do everything we can to prevent the American dream of homeownership from turning into the American nightmare of foreclosure that too many American families are experiencing.\"    \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADtpJREFUeJzt3W+MZXddx/H3x11bBCIudJ64f5htXJVFlOqwoI0lsaUsqdnlQQlLgimmyUZDFUVjFknaZHlSwPjnQdVuYA1BdIFCzMQu1oYWn5DWnf4R3NYN02XtjothYSuoYOu0Xx/MwdxO7jLn7t6Zu53f+5Xc7Dm/8/ud8z2Zzef+5tx7zqSqkCS14QcmXYAkae0Y+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGbJx0ActdccUVNT09PekyJOkF5aGHHvpGVU2t1O+SC/3p6Wnm5uYmXYYkvaAk+dc+/by8I0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDbnk7sh9IZs+cPeqH+PU7Tes+jEkrV/O9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG9Ar9JLuTnEgyn+TAkO3vTfJYki8l+XySVw5sezbJo91rdpzFS5JGs+IfRk+yAbgDeBOwABxLMltVjw10ewSYqarvJPl14EPA27tt362q1465bknSBegz098FzFfVyap6BjgC7B3sUFX3V9V3utUHgC3jLVOSNA59Qn8zcHpgfaFrO5+bgc8NrL8oyVySB5K8ddiAJPu7PnNnz57tUZIk6UKseHkHyJC2GtoxeScwA7xxoHlbVZ1JciVwX5IvV9UTz9tZ1SHgEMDMzMzQfUuSLl6fmf4CsHVgfQtwZnmnJNcB7wf2VNXT32uvqjPdvyeBLwBXXUS9kqSL0Cf0jwE7kmxPchmwD3jet3CSXAXcyVLgf32gfVOSy7vlK4CrgcEPgCVJa2jFyztVtZjkFuAeYANwuKqOJzkIzFXVLPBh4KXAp5MAPFlVe4BXAXcmeY6lN5jbl33rR5K0hvpc06eqjgJHl7XdOrB83XnGfRF4zcUUKEkaH+/IlaSGGPqS1BBDX5IaYuhLUkN6fZArSZeC6QN3r/oxTt1+w6ofY5Kc6UtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIb0Cv0ku5OcSDKf5MCQ7e9N8liSLyX5fJJXDmy7KclXutdN4yxekjSaFUM/yQbgDuAtwE7gHUl2Luv2CDBTVT8N3AV8qBv7cuA24PXALuC2JJvGV74kaRR9Zvq7gPmqOllVzwBHgL2DHarq/qr6Trf6ALClW34zcG9Vnauqp4B7gd3jKV2SNKo+ob8ZOD2wvtC1nc/NwOcucKwkaRVt7NEnQ9pqaMfkncAM8MZRxibZD+wH2LZtW4+SJEkXos9MfwHYOrC+BTizvFOS64D3A3uq6ulRxlbVoaqaqaqZqampvrVLkkbUJ/SPATuSbE9yGbAPmB3skOQq4E6WAv/rA5vuAa5Psqn7APf6rk2SNAErXt6pqsUkt7AU1huAw1V1PMlBYK6qZoEPAy8FPp0E4Mmq2lNV55J8gKU3DoCDVXVuVc5EkrSiPtf0qaqjwNFlbbcOLF/3fcYeBg5faIGSpPHxjlxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5Ia0ut5+tL3M33g7lU/xqnbb1j1Y0gtcKYvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIb0Cv0ku5OcSDKf5MCQ7dckeTjJYpIbl217Nsmj3Wt2XIVLkka34lM2k2wA7gDeBCwAx5LMVtVjA92eBN4F/O6QXXy3ql47hlolSRepz6OVdwHzVXUSIMkRYC/w/6FfVae6bc+tQo2SpDHpc3lnM3B6YH2ha+vrRUnmkjyQ5K0jVSdJGqs+M/0MaasRjrGtqs4kuRK4L8mXq+qJ5x0g2Q/sB9i2bdsIu5YkjaLPTH8B2DqwvgU40/cAVXWm+/ck8AXgqiF9DlXVTFXNTE1N9d21JGlEfUL/GLAjyfYklwH7gF7fwkmyKcnl3fIVwNUMfBYgSVpbK4Z+VS0CtwD3AI8Dn6qq40kOJtkDkOR1SRaAtwF3JjneDX8VMJfkn4D7gduXfetHkrSGev1h9Ko6Chxd1nbrwPIxli77LB/3ReA1F1mjJGlMvCNXkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDekV+kl2JzmRZD7JgSHbr0nycJLFJDcu23ZTkq90r5vGVbgkaXQrhn6SDcAdwFuAncA7kuxc1u1J4F3AXy0b+3LgNuD1wC7gtiSbLr5sSdKF6DPT3wXMV9XJqnoGOALsHexQVaeq6kvAc8vGvhm4t6rOVdVTwL3A7jHULUm6AH1CfzNwemB9oWvro9fYJPuTzCWZO3v2bM9dS5JG1Sf0M6Steu6/19iqOlRVM1U1MzU11XPXkqRR9Qn9BWDrwPoW4EzP/V/MWEnSmPUJ/WPAjiTbk1wG7ANme+7/HuD6JJu6D3Cv79okSROwYuhX1SJwC0th/Tjwqao6nuRgkj0ASV6XZAF4G3BnkuPd2HPAB1h64zgGHOzaJEkTsLFPp6o6Chxd1nbrwPIxli7dDBt7GDh8ETVKksbEO3IlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGrJx0gVI0gvB9IG7V/0Yp26/YdWP4Uxfkhpi6EtSQ7y8I12g9fLrvtriTF+SGmLoS1JDeoV+kt1JTiSZT3JgyPbLk3yy2/5gkumufTrJd5M82r3+fLzlS5JGseI1/SQbgDuANwELwLEks1X12EC3m4GnqurHkuwDPgi8vdv2RFW9dsx1S4DX1aVR9Znp7wLmq+pkVT0DHAH2LuuzF/hYt3wXcG2SjK9MSdI49An9zcDpgfWFrm1on6paBL4FvKLbtj3JI0n+IckvXmS9kqSL0Ocrm8Nm7NWzz9eAbVX1zSQ/B/xNkldX1befNzjZD+wH2LZtW4+SJEkXos9MfwHYOrC+BThzvj5JNgIvA85V1dNV9U2AqnoIeAL48eUHqKpDVTVTVTNTU1Ojn4UkqZc+oX8M2JFke5LLgH3A7LI+s8BN3fKNwH1VVUmmug+CSXIlsAM4OZ7SJUmjWvHyTlUtJrkFuAfYAByuquNJDgJzVTULfBT4eJJ54BxLbwwA1wAHkywCzwK/VlXnVuNEJEkr6/UYhqo6Chxd1nbrwPL/AG8bMu4zwGcuskZJ0ph4R64kNcTQl6SGGPqS1BAfrSxpJD764oXNmb4kNcTQl6SGrLvLO/7qKUnn50xfkhpi6EtSQwx9SWrIurumL7XAz650oZzpS1JDnOmvE878JPXhTF+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG9Ar9JLuTnEgyn+TAkO2XJ/lkt/3BJNMD297XtZ9I8ubxlS5JGtWKoZ9kA3AH8BZgJ/COJDuXdbsZeKqqfgz4I+CD3didwD7g1cBu4E+7/UmSJqDPTH8XMF9VJ6vqGeAIsHdZn73Ax7rlu4Brk6RrP1JVT1fVV4H5bn+SpAnoE/qbgdMD6wtd29A+VbUIfAt4Rc+xkqQ10udv5GZIW/Xs02csSfYD+7vV/0pyokdd43IF8I1RBuSDq1TJ2h7b8177Y4/M8x6Lkc79BXzer+zTqU/oLwBbB9a3AGfO02chyUbgZcC5nmOpqkPAoT4Fj1uSuaqamcSxJ8nzbkur5w1tn/swfS7vHAN2JNme5DKWPpidXdZnFripW74RuK+qqmvf1327ZzuwA/jH8ZQuSRrVijP9qlpMcgtwD7ABOFxVx5McBOaqahb4KPDxJPMszfD3dWOPJ/kU8BiwCLy7qp5dpXORJK0gSxPydiXZ311eaorn3ZZWzxvaPvdhmg99SWqJj2GQpIY0HforPV5iPUqyNcn9SR5PcjzJeyZd01pKsiHJI0n+dtK1rJUkP5LkriT/0v3cf37SNa2FJL/d/R//5yR/neRFk67pUtBs6Pd8vMR6tAj8TlW9CngD8O5Gzvt73gM8Puki1tifAH9XVT8J/AwNnH+SzcBvAjNV9VMsfQll32SrujQ0G/r0e7zEulNVX6uqh7vl/2QpAJq4SzrJFuAG4COTrmWtJPlh4BqWvmFHVT1TVf8x2arWzEbgh7p7h17MkHuEWtRy6Df/iIjuaahXAQ9OtpI188fA7wHPTbqQNXQlcBb4i+6y1keSvGTSRa22qvo34A+AJ4GvAd+qqr+fbFWXhpZDv9cjItarJC8FPgP8VlV9e9L1rLYkvwx8vaoemnQta2wj8LPAn1XVVcB/A+v+86skm1j6zX078KPAS5K8c7JVXRpaDv1ej4hYj5L8IEuB/4mq+uyk61kjVwN7kpxi6VLeLyX5y8mWtCYWgIWq+t5vc3ex9Caw3l0HfLWqzlbV/wKfBX5hwjVdEloO/T6Pl1h3ukdefxR4vKr+cNL1rJWqel9VbamqaZZ+1vdV1bqf+VXVvwOnk/xE13QtS3fIr3dPAm9I8uLu//y1NPABdh99Hri2Lp3v8RITLmstXA38CvDlJI92bb9fVUcnWJNW128An+gmNyeBX51wPauuqh5MchfwMEvfWHuECT3U8VLjHbmS1JCWL+9IUnMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGvJ/s/n3yZhXtA4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# load the matrix with topic proportions for each document (note that this excludes those in the dev set).\n",
    "npz = np.load(os.path.join('output', 'theta.train.npz')) \n",
    "ids = npz['ids']\n",
    "theta = npz['theta']\n",
    "n_docs, n_topics = theta.shape\n",
    "\n",
    "index = 1\n",
    "# plot the proportion of each topic in the first document\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(range(n_topics), theta[index, :])\n",
    "\n",
    "# find the original line corresponding to this document, and display the text\n",
    "print(ids[index])\n",
    "for line in lines:\n",
    "    doc = json.loads(line)\n",
    "    if doc['id'] == ids[index]:\n",
    "        print(doc['text'])\n",
    "        break\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems about right! The press release combines the topics of veterans affairs (topic 8), renewable energy (topic 1) and tax reform (topic 3). We might quibble with what the correct proportions of these topics should be, but they are cleary the dominant elements of the text, given the topics that the model has learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's plot the relative prevalance of each topics for each Senator.\n",
    "\n",
    "Because scholar split the training data into a training set and a dev set, we need to match up the output to the original senator variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAD8CAYAAACl69mTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFvFJREFUeJzt3Xu4XXV95/H3x6CGNBKsMNZHwRM74IVbhAOCA5jg5fE2IqMVJFaojszodJzamQrjZaReSjtVbFXUIqMotVwURRRrVTDiDeUEQ0K0KGBQUUQ0piomQvjOH3tFt4eTnJ1k/84mOe/X85wna//Wb/3W95eT53zyW2uftVNVSJLUwn1GXYAkaedlyEiSmjFkJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1IwhI0lqxpCRJDWzy6gLGLU99tijxsbGRl2GJO1Qli9ffntV7Tldv1kfMmNjY0xMTIy6DEnaoSS5eZB+Xi6TJDVjyEiSmjFkJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1IwhI0lqZtb/MuaqW9Yxdtpl23TsmrknDrmamXPAwr1n5DwXnXHXjJzn3uqKxWeNuoSBrF975tDHPH7hqUMfcxDnzL18aGMddfR5QxtrWy3NxU3GvXXJoibjTuZKRpLUjCEjSWrGkJEkNWPISJKaMWQkSc0YMpKkZgwZSVIzhowkqRlDRpLUzNBCJsmrk6xOsjLJiiSPG8KYy5KMD6M+SdLMG8pjZZIcATwTOLiqNiTZA7jfMMbeyjrmVNXGmT6vJGlqw1rJPAS4vao2AFTV7VX1gyT/J8nVSa5LcnaSwG9WKH+T5GtJvpXkqK591yQXdKuhC4FdN50gyVOSfCXJNUk+lGR+176mO88XgT9K8vIk3+jGuGBI85MkbYNhhcyngb26wHhnkid07e+oqkOran96gfHMvmN2qarDgD8DXte1vRS4o6oOBN4EHALQrYxeAzypqg4GJoA/7xtrfVUdWVUXAKcBj+3G+K9Dmp8kaRsMJWSq6hf0AuEU4MfAhUlOBpYk+WqSVcAxwH59h32k+3M5MNZtHw38YzfmSmBl13448BjgS0lWACcBD+8b68K+7ZXAB5O8AJjyEcBJTkkykWRi4x3rtn7CkqSBDO1R/929kGXAsi5U/gtwIDBeVd9Lcjowt++QDd2fGyfVUVMMH+AzVfX8zZz+l33bz6AXVs8CXptkv6r6nbCpqrOBswHu/5B9pjqfJGkIhrKSSfLIJPv0NS0Cru+2b+/unzx3gKGuBJZ2Y+5PL6QArgL+Q5J/3+2bl2TfKeq4D7BXVX0OeCWwOzB/G6YkSRqCYa1k5gNvT7I7vUtUN9C7dPYzYBWwBrh6gHHeBbwvyUpgBfA1gKr6cXf57fwk9+/6vgb41qTj5wD/mGQBvdXPW6vqZ9sxL0nSdhhKyFTVcuDxU+x6Tfc1uf/ivu3b6e7JVNWvgBM2c44rgEOnaB/r274TOHJrapckteNv/EuSmjFkJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1IwhI0lqxpCRJDVjyEiSmknV7H4+5Pj4eE1MTIy6DEnaoSRZXlXTfnKxKxlJUjOGjCSpGUNGktSMISNJasaQkSQ1M7SPX95RrbplHWOnXTbqMu511sw9cUbOc8DCvWfkPFO56Iy7pu+0Da5YfFaTcSdbv/bMGTnPlhy/8NSRnv+cuZc3Gfeoo89rMu72WJqLhzrerUsWDXW8zXElI0lqxpCRJDVjyEiSmjFkJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1IwhI0lqpmnIJHlYko8l+XaSG5P8fZL7JTk5yTtanluSNHrNQiZJgI8Al1TVPsC+wHzgTa3OKUm6d2m5kjkGWF9V7wOoqo3AK4AXAfOAvZJ8Ksn1SV636aAklyRZnmR1klP62n+R5G+6fZ9NcliSZUluSvKsrs9Yki8kuab7enzD+UmSptEyZPYDlvc3VNW/Ad+l92DOw4ClwCLgj5Js+hjPF1XVIcA48PIkD+rafw9Y1u37OfBG4MnAccDruz63AU+uqoOB44G3NZqbJGkALZ/CHKC20P6ZqvoJQJKPAEcCE/SC5biu717APsBPgF8Dn+raVwEbqurOJKuAsa79vsA7kiwCNtK7RHfPAnorpFMA5uy253ZMUZK0JS1XMqvprUZ+I8lu9IJjI/cMoEqyGHgScERVHQR8HZjb7b+zqjYdczewAaCq7ua3YfkK4EfAQd257zdVYVV1dlWNV9X4nHkLtnmCkqQtaxkylwPzkrwQIMkc4C3AucAdwJOT/H6SXYFnA18CFgBrq+qOJI8CDt/Kcy4AftgFzx8Dc4YyE0nSNmkWMt2q4zh691u+DXwLWA+8quvyReA8YAVwcVVN0LsctkuSlcAbgKu28rTvBE5KchW9S2W/3O6JSJK2WdNPxqyq7wH/cYpd53Zfk/tvAJ62mbHm922fPtW+qvo2cGDfrv+9lSVLkobI3/iXJDVjyEiSmjFkJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1IwhI0lqxpCRJDWT3z5zcnYaHx+viYmJUZchSTuUJMurany6fq5kJEnNGDKSpGYMGUlSM4aMJKkZQ0aS1EzTz5PZEay6ZR1jp1024+ddM/fEoY95wMK9hz7mdC46466hj3nF4rOGPmYL69eeud1jHL/w1CFUonPmXj7qEn7HUUef12zspbl4KOPcumTRUMaZjisZSVIzhowkqRlDRpLUjCEjSWrGkJEkNWPISJKaMWQkSc0YMpKkZgwZSVIzzUImSSU5r+/1Lkl+nOQT0xw3P8k/JLkxyeokVyZ53DTHfDLJ7sOqXZI0HC0fK/NLYP8ku1bVr4AnA7cMcNw5wHeAfarq7iSPAB69pQOq6unbXa0kaehaXy77Z+AZ3fbzgfM37ehWLO9LsirJyiTPSfKHwOOA11TV3QBVdVNVXdYdc0mS5d0K55S+sdYk2SPJWJJvJnlP1+fTSXZtPEdJ0ma0DpkLgBOSzAUOBL7at++1wLqqOqCqDgSuAPYDVlTVxs2M96KqOgQYB16e5EFT9NkHOKuq9gN+BjxnSHORJG2lpk9hrqqVScborWI+OWn3k4AT+vquTTLdkC9Pcly3vRe9QPnJpD7fqaoV3fZyYGzyIN0q6BSAObvtOd05JUnbaCbeXXYp8Gb6LpV1AtSkttXAQUnuUVeSxfSC6YiqOgj4OjB3ivNt6NveyBRBWlVnV9V4VY3Pmbdg0HlIkrbSTITMe4HXV9WqSe2fBv5004skD6yqG4EJ4C/TLWuS7JPkWGABsLaq7kjyKODwGahdkrQdmodMVX2/qv5+il1vBB6Y5Lok1wJLuvb/DPwBcEOSVcB7gB8AnwJ2SbISeANwVevaJUnbp9k9maqaP0XbMmBZt/0L4KQp+vwb8JLNDPu0zZxrrNu8Hdi/r/3NW1GyJGnI/I1/SVIzhowkqRlDRpLUjCEjSWrGkJEkNWPISJKaMWQkSc0YMpKkZgwZSVIzqZr8jMrZZXx8vCYmJkZdhiTtUJIsr6rx6fq5kpEkNWPISJKaMWQkSc0YMpKkZgwZSVIzzT5PZkex6pZ1jJ122TYfv2buiUOs5t7lgIV7j7qE7XLRGXeN9PxXLD5rqOOtX3vmUMcbpuMXntp0/HPmXr7Nxx519HlDrGT7Lc3Foy4BgFuXLJqR87iSkSQ1Y8hIkpoxZCRJzRgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1M23IJPlF3/bTk3w7yd5JTk/yv6Y5dlmSaT9vYIAazk3y3O0dR5I0swZeySR5IvB24KlV9d12JQ1XelyxSdIIDPTDN8lRwHuAZ1TVjVPsX5TkqiQrk3w0yQP7dr8gyZeTXJfksK7/76yCun1j3fYLu3GuTdL/0KGju3Fu2rSqSTI/yeVJrkmyKsmxXftYkm8meSdwDbDX1vylSJKGY5CQuT/wMeDZVfWvm+nzAeDUqjoQWAW8rm/f71XV44GXAe/d0omS7Ae8Gjimqg4C/kff7ocARwLPBP66a1sPHFdVBwNLgLckSbfvkcAHquqxVXXzAPOUJA3ZICFzJ/Bl4MVT7UyyANi9qj7fNb0fOLqvy/kAVXUlsFuS3bdwrmOAD1fV7d0xP+3bd0lV3V1V3wAevOn0wF8lWQl8Fnho376bq+qqzdR8SpKJJBMb71i3hXIkSdtjkJC5G3gecGiSV23DOWqK13dNOvfc7s9M0X+TDX3bm1YrS4E9gUOqahHwo76xfrnZgqrOrqrxqhqfM2/B9DOQJG2Tge7JVNUd9C5TLU3y4kn71gFru/s2AH8MfL6vy/EASY4E1nX91wAHd+0HAwu7vpcDz0vyoG7f709T2gLgtqq6M8kS4OGDzEeSNDMG/tCyqvppkqcCVya5fdLuk4B3J5kH3AT8Sd++tUm+DOwGvKhruxh4YZIVwNXAt7pzrE7yJuDzSTYCXwdO3kJZHwQ+nmQCWAFs7p6RJGkEpg2Zqprft/09frvq+Fhf+wrg8CmOXbyZMX8FPGUz+95P775Of9vJU9XU3bs5YjOl77+ZdknSDPH3RyRJzRgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1Y8hIkpoxZCRJzaRqc8+jnB3Gx8drYmJi1GVI0g4lyfKqmvaTj13JSJKaMWQkSc0YMpKkZgwZSVIzhowkqRlDRpLUzMCfjLmzWnXLOsZOu2ykNayZe+LIzn3Awr0H7nvRGXdt1dhXLD5ra8vZbuvXnjlw3+MXntqwksGdM/fyUZewRUcdfd42Hbc0Fw+5Eg3TrUsWzch5XMlIkpoxZCRJzRgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1Y8hIkpoZasgkeXCSf0pyU5LlSb6S5LitOH4syXXDrEmSNDpDC5kkAS4BrqyqR1TVIcAJwMMm9Zv1j7KRpNlimCuZY4BfV9W7NzVU1c1V9fYkJyf5UJKPA59OMj/J5UmuSbIqybF948xJ8p4kq5N8OsmuAElekuTqJNcmuTjJvK793CTvSvK5bgX1hCTvTfLNJOcOcX6SpK00zJDZD7hmC/uPAE6qqmOA9cBxVXUwsAR4S7cSAtgHOKuq9gN+Bjyna/9IVR1aVQcB3wRe3Df2A+mF3CuAjwNv7eo5IMk9ngKX5JQkE0kmNt6xbhunK0maTrMb/0nO6lYdV3dNn6mqn27aDfxVkpXAZ4GHAg/u9n2nqlZ028uBsW57/yRfSLIKWEovRDb5eFUVsAr4UVWtqqq7gdV9x/9GVZ1dVeNVNT5n3oKhzFeSdE/DvD+ymt+uOqiq/5ZkD2Cia/plX9+lwJ7AIVV1Z5I1wNxu34a+fhuBXbvtc4FnV9W1SU4GFvf123TM3ZOOvxs/zkCSRmaYK5krgLlJXtrXNm8zfRcAt3UBswR4+ADjPwD4YZL70gspSdK93ND+l19VleTZwFuTvBL4Mb3Vy6n8djWyyQeBjyeZAFYA/zrAKV4LfBW4md5lsQcMq3ZJUhtDvZRUVT+k97blqZzb1+92em8EmMr+ff3e3Lf9LuBdU5zz5L7tNZOOP3lyf0nSzPE3/iVJzRgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1Y8hIkppJ77mSs9f4+HhNTExM31GS9BtJllfV+HT9XMlIkpoxZCRJzRgykqRmDBlJUjOGjCSpGUNGktTMrP9o4lW3rGPstMuGNt6auSdu1/EHLNx7SJVM7aIz7mo6fmtXLD5rRs+3fu2ZM3q+QR2/8NRRlzAU58y9fNQlDNVRR583I+dZmou3e4xblywaQiXTcyUjSWrGkJEkNWPISJKaMWQkSc0YMpKkZgwZSVIzhowkqRlDRpLUjCEjSWqmacgk+YMkFyS5Mck3knwyyb6b6bs4ySda1iNJmlnNQiZJgI8Cy6rqD6vqMcCrgAc3Ot+sf0SOJN3btPzBvAS4s6revamhqlak52+BpwEFvLGqLuy67Jbko8AjgSuBl1XV3Ul+UVXzAZI8F3hmVZ2c5Fzgp8BjgWuS/BzYG3hE9+ffVdXbGs5RkrQFLUNmf2D5FO3/CVgEHATsAVyd5Mpu32HAY4CbgU91fT88zXn2BZ5UVRuTnA48il7APQC4Psm7qurO/gOSnAKcAjBntz23fmaSpIGM4sb/kcD5VbWxqn4EfB44tNv3taq6qao2Aud3fafzoa7/JpdV1Yaquh24jSkuz1XV2VU1XlXjc+Yt2L7ZSJI2q2XIrAYOmaI9WzimNvO6v33upD6/nPR6Q9/2Rvw4A0kamZYhcwVw/yQv2dSQ5FBgLXB8kjlJ9gSOBr7WdTksycIk9wGOB77Ytf8oyaO79uMa1ixJGqJm/8uvqkpyHPB3SU4D1gNrgD8D5gPX0luhvLKqbk3yKOArwF8DB9C78f/RbrjTgE8A3wOu646XJN3LNb2UVFU/AJ43xa6/6L76+y4Dlm1mnA8zxRsAqurkSa9Pn/R6/60oV5I0ZP7GvySpGUNGktSMISNJasaQkSQ1Y8hIkpoxZCRJzRgykqRmDBlJUjOpmvy4sNllfHy8JiYmRl2GJO1QkiyvqvHp+rmSkSQ1Y8hIkpoxZCRJzRgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1M+t/4z/Jz4HrR13HCO0B3D7qIkbI+c/e+c/mucP2z//hVbXndJ122Y4T7CyuH+TRCDurJBPO3/mPuo5RmM1zh5mbv5fLJEnNGDKSpGYMGTh71AWMmPOf3Wbz/Gfz3GGG5j/rb/xLktpxJSNJambWhEySpya5PskNSU6bYv/9k1zY7f9qkrGZr7KdAeb/50m+kWRlksuTPHwUdbYy3fz7+j03SSXZad51NMjckzyv+/6vTvJPM11jSwP82987yeeSfL379//0UdTZQpL3JrktyXWb2Z8kb+v+blYmOXjoRVTVTv8FzAFuBB4B3A+4FnjMpD4vA97dbZ8AXDjqumd4/kuAed32S2fb/Lt+DwCuBK4Cxkdd9wx+7/cBvg48sHv970Zd9wzP/2zgpd32Y4A1o657iPM/GjgYuG4z+58O/DMQ4HDgq8OuYbasZA4Dbqiqm6rq18AFwLGT+hwLvL/b/jDwxCSZwRpbmnb+VfW5qrqje3kV8LAZrrGlQb7/AG8A/i+wfiaLa2yQub8EOKuq1gJU1W0zXGNLg8y/gN267QXAD2awvqaq6krgp1vocizwgeq5Ctg9yUOGWcNsCZmHAt/re/39rm3KPlV1F7AOeNCMVNfeIPPv92J6/7vZWUw7/ySPBfaqqk/MZGEzYJDv/b7Avkm+lOSqJE+dseraG2T+pwMvSPJ94JPAf5+Z0u4VtvZnw1abLb/xP9WKZPLb6gbps6MaeG5JXgCMA09oWtHM2uL8k9wHeCtw8kwVNIMG+d7vQu+S2WJ6K9gvJNm/qn7WuLaZMMj8nw+cW1VvSXIEcF43/7vblzdyzX/uzZaVzPeBvfpeP4x7Lol/0yfJLvSWzVtaZu5IBpk/SZ4EvBp4VlVtmKHaZsJ0838AsD+wLMkaetemL91Jbv4P+m//Y1V1Z1V9h96z/PaZofpaG2T+LwYuAqiqrwBz6T3XazYY6GfD9pgtIXM1sE+ShUnuR+/G/qWT+lwKnNRtPxe4oro7YzuBaeffXS76B3oBszNdk4dp5l9V66pqj6oaq6oxeveknlVVE6Mpd6gG+bd/Cb03fpBkD3qXz26a0SrbGWT+3wWeCJDk0fRC5sczWuXoXAq8sHuX2eHAuqr64TBPMCsul1XVXUn+FPgXeu82eW9VrU7yemCiqi4F/h+9ZfIN9FYwJ4yu4uEacP5/C8wHPtS93+G7VfWskRU9RAPOf6c04Nz/BXhKkm8AG4G/qKqfjK7q4Rlw/v8TeE+SV9C7VHTyzvIfzCTn07sMukd3z+l1wH0Bqurd9O5BPR24AbgD+JOh17CT/F1Kku6FZsvlMknSCBgykqRmDBlJUjOGjCSpGUNGktSMISNJasaQkSQ1Y8hIkpr5/9s5z4IKp1n2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# load the senator variable for all the documents\n",
    "senators_df = pd.read_csv(os.path.join('tutorial', 'congress', 'train.senator.csv'), header=0, index_col=0)\n",
    "senators = senators_df.columns\n",
    "\n",
    "# pull out a subset corresponding to the ids from above\n",
    "train_subset = senators_df.loc[ids]\n",
    "n_docs, n_senators = train_subset.shape\n",
    "\n",
    "# plot the average senator-topic proportions\n",
    "fig, ax = plt.subplots()\n",
    "lefts = np.zeros(n_senators)\n",
    "for k in range(n_topics):\n",
    "    vals = []\n",
    "    for senator in senators:\n",
    "        vals.append(np.mean(theta[train_subset[senator] == 1, k]))\n",
    "\n",
    "    ax.barh(range(n_senators), vals, left=lefts)\n",
    "    lefts += np.array(vals)\n",
    "    \n",
    "ax.set_yticks(range(n_senators))\n",
    "ax.set_yticklabels(senators)\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This seems to have at least some face validity, with, for example, Amy Klobuchar writing the most about about energy/environment issues (topic 1), and Bernie Sanders writing the most about taxes and insurance (topic 3). It also appears that John McCain writes the most about the supreme court (topic 6), and Lindsay Graham devotes a lot of text to announcing grants (topic 9), but more expertise would be required to evaluate these findings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initializing with pretrained word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Emperically, we found that initializing the model with pretrained word vectors led to greater coherence in the topics. If you are interested in trying this, it is necessary to download the pretrained word2vec vectors from https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing\n",
    "\n",
    "Once this is done, and these have been saved somewhere, we can add the `--w2v` option to the model, with a path to the vector file. Here, we'll assume it's in the local directory.\n",
    "\n",
    "Note that it takes some time to load the word vectors, so we'll just try this once for illustration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator --w2v ./vectors/GoogleNews-vectors-negative300.bin\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Loading covariates from tutorial/congress/train.senator.csv\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 90 5080\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 10\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 0\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 6\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: False\n",
      "Loading word vectors\n",
      "Found embeddings for 1012 words\n",
      "Optimizing full model\n",
      "Epoch: 10 cost= 1147.214786948\n",
      "Epoch: 10; Dev perplexity = 259334511854659408208712030984883579618603079151925990842278890761964410975977487392740435859689074562890399630372673782022144.0000\n",
      "Epoch: 20 cost= 1126.638549532\n",
      "Epoch: 20; Dev perplexity = 947.9900\n",
      "Epoch: 30 cost= 1070.796553849\n",
      "Epoch: 30; Dev perplexity = 644.4405\n",
      "Epoch: 40 cost= 1059.109624056\n",
      "Epoch: 40; Dev perplexity = 607.2139\n",
      "Epoch: 50 cost= 1105.141743568\n",
      "Epoch: 50; Dev perplexity = 591.4066\n",
      "Epoch: 60 cost= 1094.154039080\n",
      "Epoch: 60; Dev perplexity = 577.8888\n",
      "Epoch: 70 cost= 1088.483892181\n",
      "Epoch: 70; Dev perplexity = 563.1722\n",
      "Epoch: 80 cost= 1139.728550580\n",
      "Epoch: 80; Dev perplexity = 551.5217\n",
      "Epoch: 90 cost= 1072.435594938\n",
      "Epoch: 90; Dev perplexity = 542.1441\n",
      "Epoch: 100 cost= 1089.287972333\n",
      "Epoch: 100; Dev perplexity = 534.1112\n",
      "Epoch: 110 cost= 1079.798876188\n",
      "Epoch: 110; Dev perplexity = 529.8079\n",
      "Epoch: 120 cost= 1089.085089835\n",
      "Epoch: 120; Dev perplexity = 515.3173\n",
      "Epoch: 130 cost= 1063.536587231\n",
      "Epoch: 130; Dev perplexity = 501.9313\n",
      "Epoch: 140 cost= 1085.577542776\n",
      "Epoch: 140; Dev perplexity = 492.9525\n",
      "Background frequencies of top words:\n",
      "obama senator bill said senate today graham washington press president\n",
      "[0.01298087 0.01290677 0.00907639 0.00863688 0.00794695 0.00677663\n",
      " 0.00639589 0.00639589 0.00609181 0.00601004]\n",
      "Topics:\n",
      "0: recovery assistance federal disaster funds damage emergency residents  / care young age statement soldiers children parents course ; sparsity=0.0000\n",
      "1: rights statement nomination following judge court position supreme  / billion costs cost fiscal amount grants care additional ; sparsity=0.0000\n",
      "2: court judge think just legal agree supreme judiciary  / announces grants grant awarded rural announced funds demint ; sparsity=0.0000\n",
      "3: statement following released vietor barack gibbs obama contact  / cost grants amount care services information provide costs ; sparsity=0.0000\n",
      "4: energy gas fuel oil climate fuels renewable production  / army county soldiers browse serve wes serving grant ; sparsity=0.0000\n",
      "5: iraq troops iraqi course war forces region terror  / grant grants rural receive county program browse illinois ; sparsity=0.0000\n",
      "6: grants departments awarded fire grant tools safety equipment  / statement immigration terror court reform legal president democratic ; sparsity=0.0000\n",
      "7: care million health funding education children research affordable  / dear statement letter concern pursuant opinion general honorable ; sparsity=0.0000\n",
      "8: soldiers veterans members care returning benefits military defense  / browse central statement grants grant public climate countries ; sparsity=0.0000\n",
      "9: announced announces press releases bishop south commerce awarded  / floor billion departments means soldiers includes fiscal military ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Covariate deviations:\n",
      "Coburn: room tom per coburn earmarks released citizen spending  / klobuchar wes gibbs petitions releases sanders bishop kevin ; sparsity=0.0000\n",
      "Graham: wes browse demint hickman relases graham bishop releases  / klobuchar obama barack letter sen amy bernie reported ; sparsity=0.0000\n",
      "Klobuchar: klobuchar minnesota serves consumer prices products amy commerce  / wes browse obama pursuant kevin gibbs press bishop ; sparsity=0.0000\n",
      "McCain: john browse mccain record air democracy colleagues simply  / obama barack wes kevin gibbs alerts petitions pursuant ; sparsity=0.0000\n",
      "Obama: pursuant brundage release illinois gibbs vietor julian petitions  / browse wes klobuchar releases bishop press graham south ; sparsity=0.0000\n",
      "Sanders: bernie sen http vermont sanders bush warming worse  / browse klobuchar releases gibbs obama south demint kevin ; sparsity=0.0000\n",
      "sparsity in covariates = 0.0000\n",
      "Dev perplexity = 484.3237\n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator --w2v ./vectors/GoogleNews-vectors-negative300.bin'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: recovery assistance federal disaster funds damage emergency residents ; sparsity=0.0000\n",
      "1: rights statement nomination following judge court position supreme ; sparsity=0.0000\n",
      "2: court judge think just legal agree supreme judiciary ; sparsity=0.0000\n",
      "3: statement following released vietor barack gibbs obama contact ; sparsity=0.0000\n",
      "4: energy gas fuel oil climate fuels renewable production ; sparsity=0.0000\n",
      "5: iraq troops iraqi course war forces region terror ; sparsity=0.0000\n",
      "6: grants departments awarded fire grant tools safety equipment ; sparsity=0.0000\n",
      "7: care million health funding education children research affordable ; sparsity=0.0000\n",
      "8: soldiers veterans members care returning benefits military defense ; sparsity=0.0000\n",
      "9: announced announces press releases bishop south commerce awarded ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "beta = np.load('output/beta.npz')['beta']\n",
    "print_top_words(beta, vocab, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the results seem quite similar, but it might be more beneficial when working with a larger vocabualry (and therefore words which occur less frequently)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding more covariates\n",
    "\n",
    "Note that we can easily include additional covarites in the same way as we did for senators. Let's try also including covariates for year and month. Just include them in a comma-separated list (again, without spaces)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator,year,month\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Loading covariates from tutorial/congress/train.senator.csv\n",
      "Loading covariates from tutorial/congress/train.year.csv\n",
      "Loading covariates from tutorial/congress/train.month.csv\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 90 5080\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 10\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 0\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 22\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: False\n",
      "Optimizing full model\n",
      "Epoch: 10 cost= 1144.751566432\n",
      "Epoch: 10; Dev perplexity = 867.6086\n",
      "Epoch: 20 cost= 1116.071242091\n",
      "Epoch: 20; Dev perplexity = 818.4166\n",
      "Epoch: 30 cost= 1086.788289115\n",
      "Epoch: 30; Dev perplexity = 745.8821\n",
      "Epoch: 40 cost= 1036.548577976\n",
      "Epoch: 40; Dev perplexity = 681.7698\n",
      "Epoch: 50 cost= 1069.456729761\n",
      "Epoch: 50; Dev perplexity = 642.0316\n",
      "Epoch: 60 cost= 1066.723419804\n",
      "Epoch: 60; Dev perplexity = 612.1673\n",
      "Epoch: 70 cost= 1047.727198249\n",
      "Epoch: 70; Dev perplexity = 585.4448\n",
      "Epoch: 80 cost= 1038.278125874\n",
      "Epoch: 80; Dev perplexity = 558.2660\n",
      "Epoch: 90 cost= 1095.816336339\n",
      "Epoch: 90; Dev perplexity = 541.2284\n",
      "Epoch: 100 cost= 1052.628236638\n",
      "Epoch: 100; Dev perplexity = 525.5809\n",
      "Epoch: 110 cost= 1060.858042331\n",
      "Epoch: 110; Dev perplexity = 510.7625\n",
      "Epoch: 120 cost= 1049.924491183\n",
      "Epoch: 120; Dev perplexity = 490.2528\n",
      "Epoch: 130 cost= 1019.492952146\n",
      "Epoch: 130; Dev perplexity = 471.8789\n",
      "Epoch: 140 cost= 1076.245264655\n",
      "Epoch: 140; Dev perplexity = 459.7113\n",
      "Background frequencies of top words:\n",
      "obama senator bill said senate today graham washington press president\n",
      "[0.01298087 0.01290677 0.00907639 0.00863688 0.00794695 0.00677663\n",
      " 0.00639589 0.00639589 0.00609181 0.00601004]\n",
      "Topics:\n",
      "0: transportation projects project water river construction infrastructure improvements  / introduced companies even called iraq country percent american ; sparsity=0.0000\n",
      "1: global warming climate change environment announces challenge announced  / funding iraq war troops soldiers appropriations president billion ; sparsity=0.0000\n",
      "2: iraq troops war iraqi forces terror course combat  / grant grants health enforcement public federal district practices ; sparsity=0.0000\n",
      "3: insurance tax care families health children americans income  / grants public grant department agency east energy justice ; sparsity=0.0000\n",
      "4: immigration judiciary legal illegal laws court supreme judge  / grant grants funds programs funding providing awarded rural ; sparsity=0.0000\n",
      "5: federal agency recovery agencies assistance residents counties lead  / economy energy class change men soldiers leadership gas ; sparsity=0.0000\n",
      "6: fire departments returning veterans care receive affairs members  / economy people economic change future debate american strong ; sparsity=0.0000\n",
      "7: programs students education college million research program agriculture  / iraq sincerely failed sent statement reform people brave ; sparsity=0.0000\n",
      "8: energy fuel renewable oil gas fuels companies alternative  / health women served care men young war soldiers ; sparsity=0.0000\n",
      "9: statement following released barack nomination release contact immediate  / provide grants providing cost funds amount percent receive ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Covariate deviations:\n",
      "Coburn: room coburn per tom citizen released spending regarding  / wes tommy relases klobuchar graham lindsey contact bishop ; sparsity=0.0000\n",
      "Graham: bishop south relases wes grant graham releases hickman  / tommy illinois vietor klobuchar barack bernie friday julian ; sparsity=0.0000\n",
      "Klobuchar: klobuchar minnesota consumer prices consumers commerce serves bipartisan  / bishop newsletters polls initiated alerts obama releases barack ; sparsity=0.0000\n",
      "McCain: mccain record browse john appropriations consequences bills democracy  / wes tommy obama barack julian graham vietor contact ; sparsity=0.0000\n",
      "Obama: illinois julian obama tommy barack pursuant gibbs vietor  / bishop south klobuchar lindsey browse carolina releases press ; sparsity=0.0000\n",
      "Sanders: bernie vermont http sanders visit sen read news  / amy bishop labolt releases obama newsletters polls browse ; sparsity=0.0000\n",
      "2005: julian gibbs tommy vietor green announces supreme court  / pursuant amy initiated ortiz newsletters electronic alerts brundage ; sparsity=0.0000\n",
      "2006: tommy vietor gibbs julian terrorists robert credit terror  / pursuant alerts amy polls petitions ortiz opinion brundage ; sparsity=0.0000\n",
      "2007: labolt ben brundage iraq amy consequences troops sanders  / gibbs julian vietor tommy robert ortiz pursuant alerts ; sparsity=0.0000\n",
      "2008: newsletters initiated michael ortiz petitions polls alerts opinion  / julian gibbs tommy labolt ben vietor bishop robert ; sparsity=0.0000\n",
      "Apr: april location prices price temporary drug lower clearly  / june september august july essential seen terror november ; sparsity=0.0000\n",
      "Aug: august receive grant purchase transportation writing announce insurance  / decades continues clinton creating dollars expected away proud ; sparsity=0.0000\n",
      "Dec: december pursuant petitions newsletters electronic initiated alerts polls  / june labolt ben richard company labor march announces ; sparsity=0.0000\n",
      "Feb: february cut budget bipartisan message open eliminate responsible  / york established june september april veto along organizations ; sparsity=0.0000\n",
      "Jan: january economic legislative reduction sources things essential associated  / estimated october july june september city april expected ; sparsity=0.0000\n",
      "Jul: july truly improving debt freedom quality district goes  / brundage november october april september location respond class ; sparsity=0.0000\n",
      "Jun: june initiative choice union chicago amendment elected investment  / october august december amy ensuring veto september april ; sparsity=0.0000\n",
      "Mar: march rates budget resolution conditions alone subject billions  / june september brundage look august authorization july october ; sparsity=0.0000\n",
      "May: helps committed illegal immigration domestic nothing credit comprehensive  / september june july august october location follow november ; sparsity=0.0000\n",
      "Nov: november petitions pursuant polls brundage newsletters initiated alerts  / labolt june close designed establish approach even thousands ; sparsity=0.0000\n",
      "Oct: october brundage amy private reported clinton specifically civil  / labolt june ben january april designed illinois ortiz ; sparsity=0.0000\n",
      "Sep: september amy brundage chief terror ahead something justice  / labolt june encourage initiated january reform november december ; sparsity=0.0000\n",
      "sparsity in covariates = 0.0000\n",
      "Dev perplexity = 452.5664\n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator,year,month'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if we load the covariate vectors, we will also see some temporal patterns in word frequencies. The ones here don't seem all that compelling, but perhaps more data is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coburn: room coburn per tom citizen released spending regarding ; sparsity=0.0000\n",
      "Graham: bishop south relases wes grant graham releases hickman ; sparsity=0.0000\n",
      "Klobuchar: klobuchar minnesota consumer prices consumers commerce serves bipartisan ; sparsity=0.0000\n",
      "McCain: mccain record browse john appropriations consequences bills democracy ; sparsity=0.0000\n",
      "Obama: illinois julian obama tommy barack pursuant gibbs vietor ; sparsity=0.0000\n",
      "Sanders: bernie vermont http sanders visit sen read news ; sparsity=0.0000\n",
      "2005: julian gibbs tommy vietor green announces supreme court ; sparsity=0.0000\n",
      "2006: tommy vietor gibbs julian terrorists robert credit terror ; sparsity=0.0000\n",
      "2007: labolt ben brundage iraq amy consequences troops sanders ; sparsity=0.0000\n",
      "2008: newsletters initiated michael ortiz petitions polls alerts opinion ; sparsity=0.0000\n",
      "Apr: april location prices price temporary drug lower clearly ; sparsity=0.0000\n",
      "Aug: august receive grant purchase transportation writing announce insurance ; sparsity=0.0000\n",
      "Dec: december pursuant petitions newsletters electronic initiated alerts polls ; sparsity=0.0000\n",
      "Feb: february cut budget bipartisan message open eliminate responsible ; sparsity=0.0000\n",
      "Jan: january economic legislative reduction sources things essential associated ; sparsity=0.0000\n",
      "Jul: july truly improving debt freedom quality district goes ; sparsity=0.0000\n",
      "Jun: june initiative choice union chicago amendment elected investment ; sparsity=0.0000\n",
      "Mar: march rates budget resolution conditions alone subject billions ; sparsity=0.0000\n",
      "May: helps committed illegal immigration domestic nothing credit comprehensive ; sparsity=0.0000\n",
      "Nov: november petitions pursuant polls brundage newsletters initiated alerts ; sparsity=0.0000\n",
      "Oct: october brundage amy private reported clinton specifically civil ; sparsity=0.0000\n",
      "Sep: september amy brundage chief terror ahead something justice ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "topic_covars = np.load(os.path.join('output', 'beta_c.npz'))\n",
    "weights = topic_covars['beta']\n",
    "names = topic_covars['names']\n",
    "print_top_words(weights, vocab, topic_names=names, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interactions\n",
    "\n",
    "Alternatively, we can include interactions between covariates and topics.\n",
    "\n",
    "Here, let's try using a covariate for party membership, rather than for each Senator, and include interactions between topics and party. To do this, just include the `--interactions` options. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 99 --topic-covars party --interactions\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Loading covariates from tutorial/congress/train.party.csv\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 88 5075\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 10\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 0\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 2\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: True\n",
      "Optimizing full model\n",
      "Epoch: 10 cost= 1154.104391691\n",
      "Epoch: 10; Dev perplexity = 1038.3428\n",
      "Epoch: 20 cost= 1109.740894155\n",
      "Epoch: 20; Dev perplexity = 954.6553\n",
      "Epoch: 30 cost= 1104.557029502\n",
      "Epoch: 30; Dev perplexity = 778.1132\n",
      "Epoch: 40 cost= 1070.655842553\n",
      "Epoch: 40; Dev perplexity = 702.9757\n",
      "Epoch: 50 cost= 1089.706998479\n",
      "Epoch: 50; Dev perplexity = 669.3897\n",
      "Epoch: 60 cost= 1074.122579348\n",
      "Epoch: 60; Dev perplexity = 630.9227\n",
      "Epoch: 70 cost= 1053.061446973\n",
      "Epoch: 70; Dev perplexity = 600.0517\n",
      "Epoch: 80 cost= 1070.874946475\n",
      "Epoch: 80; Dev perplexity = 577.6437\n",
      "Epoch: 90 cost= 1053.263297198\n",
      "Epoch: 90; Dev perplexity = 556.8440\n",
      "Epoch: 100 cost= 1068.774790924\n",
      "Epoch: 100; Dev perplexity = 540.7807\n",
      "Epoch: 110 cost= 1076.868975680\n",
      "Epoch: 110; Dev perplexity = 524.2459\n",
      "Epoch: 120 cost= 1090.976070942\n",
      "Epoch: 120; Dev perplexity = 503.9967\n",
      "Epoch: 130 cost= 1116.502120255\n",
      "Epoch: 130; Dev perplexity = 486.8766\n",
      "Epoch: 140 cost= 1092.787315829\n",
      "Epoch: 140; Dev perplexity = 474.5702\n",
      "Background frequencies of top words:\n",
      "senator obama bill said senate today washington graham press president\n",
      "[0.01296816 0.01271263 0.00941629 0.00862415 0.00805176 0.00681244\n",
      " 0.00638571 0.00630138 0.00597941 0.00590275]\n",
      "Topics:\n",
      "0: iraq armed mission terrorists war troops forces reports  / bishop hickman kevin south wes releases lindsey environmental ; sparsity=0.0000\n",
      "1: durbin announce respond residents awarded supports assistance department  / american people just like change take mccain see ; sparsity=0.0000\n",
      "2: earmarks bills income insurance costs provisions fund health  / graham bishop releases kevin contact hickman lindsey wes ; sparsity=0.0000\n",
      "3: armed veterans families army back returning men duty  / alerts newsletters initiated ortiz polls pursuant petitions immediate ; sparsity=0.0000\n",
      "4: coburn room sanders citizen lower bernie tom vermont  / graham bishop kevin hickman releases alerts relases immediate ; sparsity=0.0000\n",
      "5: announces announced bishop wes awarded lead relases press  / back away mccain fiscal families president hard real ; sparsity=0.0000\n",
      "6: court judiciary statement nomination supreme ortiz newsletters judge  / million additional program percent services sanders provide cost ; sparsity=0.0000\n",
      "7: fuel fuels renewable production sources oil illegal alternative  / services health ortiz demint funds office alerts newsletters ; sparsity=0.0000\n",
      "8: democracy troops iraq freedom war region resolution statement  / department million programs federal health pay provide enforcement ; sparsity=0.0000\n",
      "9: river land project development facility klobuchar minnesota water  / americans obama president newsletters polls far millions barack ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Covariate deviations:\n",
      "D: brundage efficient tuesday oil veterans vietor fuels children  / graham bishop lindsey browse kevin relases demint hickman ; sparsity=0.0000\n",
      "R: releases demint graham mccain kevin grant browse grants  / tommy ortiz obama alerts julian release immediate pursuant ; sparsity=0.0000\n",
      "sparsity in covariates = 0.0000\n",
      "Covariate interactions\n",
      "(20, 1021)\n",
      "0:D: reports defense service armed review members military receiving  / graham land browse press supports reduction mccain hickman ; sparsity=0.0000\n",
      "0:R: terrorists rather beginning mission iraq just iraqi region  / vietor contact brundage pursuant health julian friday alerts ; sparsity=0.0000\n",
      "1:D: durbin illinois newsletters request dear affected ortiz dick  / graham hickman bishop kevin press lindsey awarded wes ; sparsity=0.0000\n",
      "1:R: grants amount awarded supports announce training kind demint  / immediate soon find president states learn keep led ; sparsity=0.0000\n",
      "2:D: low program insurance families income affordable housing relief  / graham force press decision browse earmarks armed required ; sparsity=0.0010\n",
      "2:R: earmarks rather coburn unfortunately largest included projects contains  / graham contact bishop kevin relases date wes strengthen ; sparsity=0.0000\n",
      "3:D: returning veterans minnesota klobuchar affairs soldiers field lives  / graham kevin hickman environmental mccain bishop alerts lindsey ; sparsity=0.0000\n",
      "3:R: armed defense terrorism freedom terror country force part  / friday returning vietor caused julian obama reports grant ; sparsity=0.0000\n",
      "4:D: sanders bernie vermont told http lower paying news  / immediate newsletters ensure graham release date electronic ortiz ; sparsity=0.0000\n",
      "4:R: coburn room citizen per tom spending released agencies  / bishop graham kevin date contact college lindsey carolina ; sparsity=0.0000\n",
      "5:D: consumer lead safety dear sincerely text commission consumers  / browse graham bishop relases lindsey south press carolina ; sparsity=0.0000\n",
      "5:R: hickman lindsey kevin awarded bishop announces record wes  / immediate now obama people measures away matter just ; sparsity=0.0000\n",
      "6:D: pursuant polls newsletters alerts petitions opinion ortiz barack  / graham browse relases current hickman percent additional press ; sparsity=0.0000\n",
      "6:R: court supreme judiciary nomination judge man experience position  / immediate million initiated vietor newsletters pursuant ortiz month ; sparsity=0.0000\n",
      "7:D: fuel fuels renewable energy sources gas oil reduce  / graham armed human kevin bishop lindsey health judiciary ; sparsity=0.0000\n",
      "7:R: legal immigration get illegal judiciary period back wes  / vietor directly julian jim per funds obama initiative ; sparsity=0.0000\n",
      "8:D: troops war bring terrorism iraq failed change united  / graham browse mccain relases lindsey bishop services press ; sparsity=0.0000\n",
      "8:R: democracy hickman wes terror bishop press kevin record  / vietor million gibbs cost newsletters initiated electronic julian ; sparsity=0.0000\n",
      "9:D: minnesota klobuchar cities river funding works residents new  / graham date browse hickman press relases kevin contact ; sparsity=0.0000\n",
      "9:R: development jim south carolina demint million contact announce  / just far people congress day klobuchar things decisions ; sparsity=0.0000\n",
      "sparsity in covariate interactions = 0.0000\n",
      "Dev perplexity = 466.3883\n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 10 --epochs 150 --dev-folds 10 --seed 99 --topic-covars party --interactions'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's look at the vectors learned for each party, then the topics, then the interactions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "D: brundage efficient tuesday oil veterans vietor fuels children ; sparsity=0.0000\n",
      "R: releases demint graham mccain kevin grant browse grants ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "topic_covars = np.load(os.path.join('output', 'beta_c.npz'))\n",
    "weights = topic_covars['beta']\n",
    "names = topic_covars['names']\n",
    "print_top_words(weights, vocab, topic_names=names, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: iraq armed mission terrorists war troops forces reports ; sparsity=0.0000\n",
      "1: durbin announce respond residents awarded supports assistance department ; sparsity=0.0000\n",
      "2: earmarks bills income insurance costs provisions fund health ; sparsity=0.0000\n",
      "3: armed veterans families army back returning men duty ; sparsity=0.0000\n",
      "4: coburn room sanders citizen lower bernie tom vermont ; sparsity=0.0000\n",
      "5: announces announced bishop wes awarded lead relases press ; sparsity=0.0000\n",
      "6: court judiciary statement nomination supreme ortiz newsletters judge ; sparsity=0.0000\n",
      "7: fuel fuels renewable production sources oil illegal alternative ; sparsity=0.0000\n",
      "8: democracy troops iraq freedom war region resolution statement ; sparsity=0.0000\n",
      "9: river land project development facility klobuchar minnesota water ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "beta = np.load(os.path.join('output', 'beta.npz'))['beta']\n",
    "print_top_words(beta, vocab, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:D: reports defense service armed review members military receiving ; sparsity=0.0000\n",
      "0:R: terrorists rather beginning mission iraq just iraqi region ; sparsity=0.0000\n",
      "1:D: durbin illinois newsletters request dear affected ortiz dick ; sparsity=0.0000\n",
      "1:R: grants amount awarded supports announce training kind demint ; sparsity=0.0000\n",
      "2:D: low program insurance families income affordable housing relief ; sparsity=0.0010\n",
      "2:R: earmarks rather coburn unfortunately largest included projects contains ; sparsity=0.0000\n",
      "3:D: returning veterans minnesota klobuchar affairs soldiers field lives ; sparsity=0.0000\n",
      "3:R: armed defense terrorism freedom terror country force part ; sparsity=0.0000\n",
      "4:D: sanders bernie vermont told http lower paying news ; sparsity=0.0000\n",
      "4:R: coburn room citizen per tom spending released agencies ; sparsity=0.0000\n",
      "5:D: consumer lead safety dear sincerely text commission consumers ; sparsity=0.0000\n",
      "5:R: hickman lindsey kevin awarded bishop announces record wes ; sparsity=0.0000\n",
      "6:D: pursuant polls newsletters alerts petitions opinion ortiz barack ; sparsity=0.0000\n",
      "6:R: court supreme judiciary nomination judge man experience position ; sparsity=0.0000\n",
      "7:D: fuel fuels renewable energy sources gas oil reduce ; sparsity=0.0000\n",
      "7:R: legal immigration get illegal judiciary period back wes ; sparsity=0.0000\n",
      "8:D: troops war bring terrorism iraq failed change united ; sparsity=0.0000\n",
      "8:R: democracy hickman wes terror bishop press kevin record ; sparsity=0.0000\n",
      "9:D: minnesota klobuchar cities river funding works residents new ; sparsity=0.0000\n",
      "9:R: development jim south carolina demint million contact announce ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "interactions = np.load(os.path.join('output', 'beta_ci.npz'))\n",
    "weights = interactions['beta']\n",
    "names = topic_covars['names']\n",
    "names = [str(k) + ':' + c for k in range(10) for c in names]\n",
    "print_top_words(weights, vocab, topic_names=names, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic topics here are now perhaps less good, but the interaction do seem to capture something about how the different parties talk about certain issues. For example, for topic 0 (Iraq), The Democrat version emphasizes reports and defense, whereas the Republican version is more about terrorsm. More data (from all the senators) would give us a much better approxaimation of this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to covariates, we can also introduce labels, which are predicted from the topics that are learned. Although they only introduce a subtle influence on the topic, we are effectively learning a classifier and a topic model simultaneously.\n",
    "\n",
    "Here, let's try predicting the party of a press release from the topics. We'll also try using slightly more topics this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python run_scholar.py tutorial/congress/ -k 12 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator,month --label party\n",
      "Loading data\n",
      "Loaded 2483 documents with 1021 features\n",
      "Found 2483 non-empty documents\n",
      "Loading labels from tutorial/congress/train.party.csv\n",
      "Found 2 labels\n",
      "Loading covariates from tutorial/congress/train.senator.csv\n",
      "Loading covariates from tutorial/congress/train.month.csv\n",
      "Train label proportions: [0.54007249 0.45992751]\n",
      "Computing background frequencies\n",
      "Min/max word counts in training data: 90 5080\n",
      "Network architecture:\n",
      "embedding_dim: 300\n",
      "n_topics: 12\n",
      "vocab_size: 1021\n",
      "label_type: None\n",
      "n_labels: 2\n",
      "n_prior_covars: 0\n",
      "n_topic_covars: 18\n",
      "l1_beta_reg: 0.0\n",
      "l1_beta_c_reg: 0.0\n",
      "l1_beta_ci_reg: 0.0\n",
      "l2_prior_reg: 0.0\n",
      "classifier_layers: 1\n",
      "use_interactions: False\n",
      "Optimizing full model\n",
      "Epoch: 10 ; cost = 1147.348687430 ; training accuracy (noisy) = 0.666666667\n",
      "Epoch: 10; Dev perplexity = 937.0897; Dev accuracy = 0.7621\n",
      "Epoch: 20 ; cost = 1115.941902178 ; training accuracy (noisy) = 0.666219239\n",
      "Epoch: 20; Dev perplexity = 867.0816; Dev accuracy = 0.7500\n",
      "Epoch: 30 ; cost = 1086.522339686 ; training accuracy (noisy) = 0.661297539\n",
      "Epoch: 30; Dev perplexity = 764.8252; Dev accuracy = 0.7379\n",
      "Epoch: 40 ; cost = 1039.726731815 ; training accuracy (noisy) = 0.680089485\n",
      "Epoch: 40; Dev perplexity = 712.4887; Dev accuracy = 0.7298\n",
      "Epoch: 50 ; cost = 1071.697328981 ; training accuracy (noisy) = 0.680984340\n",
      "Epoch: 50; Dev perplexity = 666.2135; Dev accuracy = 0.7460\n",
      "Epoch: 60 ; cost = 1069.788089215 ; training accuracy (noisy) = 0.669351230\n",
      "Epoch: 60; Dev perplexity = 632.0942; Dev accuracy = 0.7621\n",
      "Epoch: 70 ; cost = 1051.241861979 ; training accuracy (noisy) = 0.709619687\n",
      "Epoch: 70; Dev perplexity = 603.9479; Dev accuracy = 0.7581\n",
      "Epoch: 80 ; cost = 1041.261453309 ; training accuracy (noisy) = 0.710961969\n",
      "Epoch: 80; Dev perplexity = 582.9144; Dev accuracy = 0.7702\n",
      "Epoch: 90 ; cost = 1099.019237407 ; training accuracy (noisy) = 0.694854586\n",
      "Epoch: 90; Dev perplexity = 564.5949; Dev accuracy = 0.7823\n",
      "Epoch: 100 ; cost = 1056.024527667 ; training accuracy (noisy) = 0.713646532\n",
      "Epoch: 100; Dev perplexity = 546.9456; Dev accuracy = 0.7863\n",
      "Epoch: 110 ; cost = 1065.826888458 ; training accuracy (noisy) = 0.739149888\n",
      "Epoch: 110; Dev perplexity = 534.9078; Dev accuracy = 0.7944\n",
      "Epoch: 120 ; cost = 1055.295180762 ; training accuracy (noisy) = 0.715883669\n",
      "Epoch: 120; Dev perplexity = 508.3234; Dev accuracy = 0.8105\n",
      "Epoch: 130 ; cost = 1025.042934887 ; training accuracy (noisy) = 0.727964206\n",
      "Epoch: 130; Dev perplexity = 488.5644; Dev accuracy = 0.8065\n",
      "Epoch: 140 ; cost = 1081.518232444 ; training accuracy (noisy) = 0.708724832\n",
      "Epoch: 140; Dev perplexity = 473.1687; Dev accuracy = 0.7944\n",
      "Background frequencies of top words:\n",
      "obama senator bill said senate today graham washington press president\n",
      "[0.01298087 0.01290677 0.00907639 0.00863688 0.00794695 0.00677663\n",
      " 0.00639589 0.00639589 0.00609181 0.00601004]\n",
      "Topics:\n",
      "0: transparency taxpayers accountability taxpayer financial dollars companies federal  / great war infrastructure bush region human terror brave ; sparsity=0.0000\n",
      "1: lead products consumer commission trade safety dangerous free  / funds awarded receive training average cost programs grant ; sparsity=0.0000\n",
      "2: health care insurance costs tax income priorities americans  / general follow commitment durbin security staff browse conduct ; sparsity=0.0000\n",
      "3: projects recovery disaster assistance damage counties infrastructure funds  / care serve health policies comprehensive strengthen laws americans ; sparsity=0.0000\n",
      "4: veterans returning benefits affairs defense members guard duty  / interest people companies economy industry citizens businesses market ; sparsity=0.0000\n",
      "5: announces relases record announced press browse releases wes  / amendment cost bill percent without additional provisions billion ; sparsity=0.0000\n",
      "6: grants awarded training departments grant announce equipment receive  / issue senate reform vote legislation provisions passed member ; sparsity=0.0000\n",
      "7: supreme court judge don judiciary values let legal  / funds additional bill awarded funding provide million programs ; sparsity=0.0000\n",
      "8: energy gas renewable oil environment vehicles fuel fuels  / election office ortiz ask polls days statement reports ; sparsity=0.0000\n",
      "9: immigration troops iraq illegal border security war failed  / federal health used access announced awarded grant project ; sparsity=0.0000\n",
      "10: sincerely letter dear text workers writing sent hearing  / bill ever funds introduce spending troops generation amendment ; sparsity=0.0000\n",
      "11: statement pursuant newsletters ortiz alerts polls petitions released  / percent health average based care increase problem credit ; sparsity=0.0000\n",
      "sparsity in topics = 0.0000\n",
      "Covariate deviations:\n",
      "Coburn: citizen coburn tom room per released earmarks spending  / releases browse carolina klobuchar obama graham sanders hickman ; sparsity=0.0000\n",
      "Graham: kevin releases carolina browse relases hickman graham south  / tommy obama release thursday gibbs klobuchar green barack ; sparsity=0.0000\n",
      "Klobuchar: klobuchar minnesota commerce amy secured beyond serves consumers  / releases carolina browse contact gibbs tommy press polls ; sparsity=0.0000\n",
      "McCain: mccain browse john though freedom record simply democracy  / tommy contact gibbs graham klobuchar hickman vietor wes ; sparsity=0.0000\n",
      "Obama: gibbs tommy obama labolt release polls vietor julian  / releases carolina relases press record lindsey graham sanders ; sparsity=0.0000\n",
      "Sanders: vermont http sanders sen bernie read paying food  / alerts obama relases browse contact barack tommy releases ; sparsity=0.0000\n",
      "Apr: april location temporary price prices drug lower clearly  / september august july june brundage amy gibbs terror ; sparsity=0.0000\n",
      "Aug: august grant receive transportation purchase monday illinois cities  / petitions expected clinton initiated brundage election continues join ; sparsity=0.0000\n",
      "Dec: december farm farmers newsletters alerts contains provision electronic  / taking labolt june applaud ben offer august company ; sparsity=0.0000\n",
      "Feb: february cut message budget open bipartisan affected responsible  / along august july september york brundage received june ; sparsity=0.0000\n",
      "Jan: january economic class pleased judiciary judge essential let  / july october september estimated labolt city amy brundage ; sparsity=0.0000\n",
      "Jul: july goes truly freedom quality improving strengthen debt  / brundage november october february amy september location august ; sparsity=0.0000\n",
      "Jun: june union initiative laws investment illinois system existing  / august october amy brundage march february september january ; sparsity=0.0000\n",
      "Mar: march billions subject budget rates alone resolution paid  / june brundage july september october august look supreme ; sparsity=0.0000\n",
      "May: immigration committed aid helps alternative months reform nothing  / september july brundage june increases august amy follow ; sparsity=0.0000\n",
      "Nov: november brundage win commitment projects republican petitions veto  / july ortiz labolt noted thousands march later ben ; sparsity=0.0000\n",
      "Oct: october brundage amy specifically civil human reported days  / labolt polls ortiz electronic period pursuant election initiated ; sparsity=0.0000\n",
      "Sep: september chief brundage terror something terrorists justice getting  / labolt alerts ben ortiz petitions electronic pursuant january ; sparsity=0.0000\n",
      "sparsity in covariates = 0.0000\n",
      "Dev perplexity = 464.6240\n",
      "Predicting labels\n",
      "train accuracy on labels = 0.8018\n",
      "dev accuracy on labels = 0.7823\n",
      "Label probabilities based on topics\n",
      "Labels: D R\n",
      "0: 0.5540 0.4460 \n",
      "1: 0.6336 0.3664 \n",
      "2: 0.7252 0.2748 \n",
      "3: 0.6767 0.3233 \n",
      "4: 0.8098 0.1902 \n",
      "5: 0.0099 0.9901 \n",
      "6: 0.0651 0.9349 \n",
      "7: 0.1596 0.8404 \n",
      "8: 0.8844 0.1156 \n",
      "9: 0.2040 0.7960 \n",
      "10: 0.9209 0.0791 \n",
      "11: 0.9858 0.0142 \n",
      "Saving document representations\n"
     ]
    }
   ],
   "source": [
    "script = 'run_scholar.py'\n",
    "args = 'tutorial/congress/ -k 12 --epochs 150 --dev-folds 10 --seed 42 --topic-covars senator,month --label party'\n",
    "print(\"python\", script, args)\n",
    "run_scholar.main(args.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's look at the topics as usual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0: transparency taxpayers accountability taxpayer financial dollars companies federal ; sparsity=0.0000\n",
      "1: lead products consumer commission trade safety dangerous free ; sparsity=0.0000\n",
      "2: health care insurance costs tax income priorities americans ; sparsity=0.0000\n",
      "3: projects recovery disaster assistance damage counties infrastructure funds ; sparsity=0.0000\n",
      "4: veterans returning benefits affairs defense members guard duty ; sparsity=0.0000\n",
      "5: announces relases record announced press browse releases wes ; sparsity=0.0000\n",
      "6: grants awarded training departments grant announce equipment receive ; sparsity=0.0000\n",
      "7: supreme court judge don judiciary values let legal ; sparsity=0.0000\n",
      "8: energy gas renewable oil environment vehicles fuel fuels ; sparsity=0.0000\n",
      "9: immigration troops iraq illegal border security war failed ; sparsity=0.0000\n",
      "10: sincerely letter dear text workers writing sent hearing ; sparsity=0.0000\n",
      "11: statement pursuant newsletters ortiz alerts polls petitions released ; sparsity=0.0000\n"
     ]
    }
   ],
   "source": [
    "beta = np.load(os.path.join('output', 'beta.npz'))['beta']\n",
    "print_top_words(beta, vocab, n_pos=8, n_neg=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at which topics predict Democrat vs Republican."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels: D R\n",
      "0: 0.5540 0.4460 \n",
      "1: 0.6336 0.3664 \n",
      "2: 0.7252 0.2748 \n",
      "3: 0.6767 0.3233 \n",
      "4: 0.8098 0.1902 \n",
      "5: 0.0099 0.9901 \n",
      "6: 0.0651 0.9349 \n",
      "7: 0.1596 0.8404 \n",
      "8: 0.8844 0.1156 \n",
      "9: 0.2040 0.7960 \n",
      "10: 0.9209 0.0791 \n",
      "11: 0.9858 0.0142 \n"
     ]
    }
   ],
   "source": [
    "npz = np.load('output/topics_to_labels.npz')\n",
    "probs = npz['probs']\n",
    "label_names = npz['label']\n",
    "n_topics, n_labels = probs.shape\n",
    "print(\"Labels:\", ' '.join([name for name in label_names]))\n",
    "for k in range(n_topics):\n",
    "    output = str(k) + ': '\n",
    "    for i in range(n_labels):\n",
    "        output += '%.4f ' % probs[k, i]\n",
    "    print(output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These numbers show the probability (according to the model) that a document entirely about a single topic is from a Democrat vs a Republican. Of course, in practice, most documents will be represented as a mixture of topics, and both parties talk about all topics to some degree.\n",
    "\n",
    "More expertise would be useful in trying to interpret whether the model is doing something reasonable here, but at least some parts seem to line up (e.g. Democrats talking more about health care (topic 2) and Republicans talking more about illegal immigrtion / Iraq (topic 9), but others seem to be overfitting to particular people (e.g. topics 5, 6, and 11). As above, using data from more Senators would be very useful here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ultimately, exploration of different models is requried to figure out what is best for your application. Also, remember than trying a different random seed will give you different results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}