README for Termite, a topic model visualization tool.

---------------
  INFORMATION
---------------
Termite is a visualization tool for inspecting the output of statistical
topic models based on the techniques described in the following publication:

Termite: Visualization Techniques for Assessing Textual Topic Models
Jason Chuang, Christopher D. Manning, Jeffrey Heer
Computer Science Dept, Stanford University
http://vis.stanford.edu/papers/termite

This tool is developed by:
    * Jason Chuang <jcchuang@cs.stanford.edu>
    * Ashley Jin <ashpjin@cs.stanford.edu>

and is distributed under the BSD-3 license.

-----------
  LICENSE
-----------
Copyright (c) 2013, Leland Stanford Junior University
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the <organization> nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

----------------------
  ADDITIONAL CREDITS
----------------------
Termite requires the use of the following libraries and tools.
We thank their respective authors for developing and distributing the tools.

Mallet: Machine learning for language toolkit
    Project website: http://mallet.cs.umass.edu
    Developed by Andrew McCallum, et al.
    Distributed under a CPL license: lib/LICENSE-mallet

STMT: Stanford topic modeling toolbox
    Project website: http://nlp.stanford.edu/software/tmt
    Developed by Daniel Ramage, et al.
    Distributed under a GNU license: lib/LICENSE-stmt

D3 javascript visualization library
    Project website: http://d3js.org
    Developed by Mike Bostock, et al.
    Distributed under a BSD license: lib/LICENSE-d3

Google closure javascript compiler
    Project website: https://developers.google.com/closure/compiler/
    Developed by Google engineers
    Distributed under an Apache license: lib/LICENSE-closure-compiler

Backbone
    Project website: http://backbonejs.org
    Developed by Jeremy Ashkenas, DocumentCloud Inc.
    Distributed under a MIT license: lib/LICENSE-backbone

Underscore
    Project website: http://underscorejs.org
    Developed by Jeremy Ashkenas, DocumentCloud Inc.
    Distributed under a MIT license: lib/LICENSE-underscore

jQuery
    Project website: http://jquery.com
    Developed by the jQuery Foundation
    Distributed under a MIT license: lib/LICENSE-jquery

html5slider
    Project website: https://github.com/fryn/html5slider
    Developed by Frank Yan
    Distributed under an MIT license: http://opensource.org/licenses/MIT

------------------
  ONE-TIME SETUP
------------------
Run the setup script to fetch the following tools, libraries, and prepare all javascript
files needed by Termite. This script only needs to be run once when Termite is first
downloaded onto a new machine.
    >> ./setup.sh

Libraries fetched include:
    * mallet
    * stmt
    * closure-compiler.js
    * d3.v3.js
    * jquery.js
    * underscore.js
    * backbone.js
    * html5slider.js

The script creates a minified version of all javascript files.

-----------------------
  BUILD A TOPIC MODEL
-----------------------
Building a topic model in Termite requires running a single python script. The script reads
in an input text corpus, and produces an output folder whose content can be inspected 
using a web browser, described in the next section.

Customize configuration file with the following information. A sample configuration
file can be found in 'example.cfg'
    [Corpus]       path to text corpus
    [TopicModel]   directory for holding topic model outputs
    [TopicModel]   number of topics to train
    [TopicModel]   topic model (either mallet or stmt)
    [Termite]      number of terms to seriate
    [Termite]      path to save Termite-internal working files
    
Process the text corpus, and build a topic model by running the execution script.
Execution time will vary depending on the size of the corpus.
    >> ./execute.py <your_config_file.cfg>

The execution script calls in order: 
    1. pipeline/tokenize.py                  Tokenize the text corpus
    2. pipeline/train_mallet.py              Train a topic model using MALLET
       pipeline/train_stmt.py                or STMT.
    3. pipeline/compute_saliency.py          Compute term saliency
    4. pipeline/compute_similarity.py        Compute term similarity
    5. pipeline/compute_seriation.py:        Seriates terms
    6. pipeline/prepare_data_for_client.py:  Generates datafiles for client
    7. pipeline/prepare_vis_for_client.py:   Copies necessary scripts for client

---------------------------- 
  VIEW TOPIC MODEL OUTPUTS
----------------------------
You are now ready to visualize the topic model outputs! Termite's output can be viewed in
a web browser. To view the files locally (on your own computer), you need to set up a local
web server. Alternatively, you may copy the output folder to a web server to publish the results.

Termite outputs are stored in the 'public_html' subfolder within the output directory.

To set up a local webserver:
    1. Change into output directory (specified in the configuration file)
       >> cd <output_folder>/public_html
    2. Start a local server using python
       >> ./web.sh
    3. Open http://localhost:8888 in a modern web browser (Chrome, Safari, Firefox, or Opera)
       to view a visualization of the model outputs.

To publish the results on a webserver:
    1. Copy public_html directory to your remote server.

-----------------------------
  TOPIC MODEL VISUALIZATION
-----------------------------
The visualization should consist of a matrix view with a frequency bar view to the right.
The top right contains user controls, two slider bars and one input text box. User
controls are explained in more detail in the 'VISUALIZATION USE CASES' section.

When the mouse is placed over terms, topics, circles, or bars in the matrix or frequency 
bar view, the associated term, topic, bar, and circles will be highlighted in both views.

---------------------------
  VISUALIZATION USE CASES
---------------------------
The Termite visualization tool has a set of user controls in the top right section of the
webpage. The user may specify
    * the number of terms with highest affinity to show
    * the number of terms with highest saliency to show
    * always display specific terms
    * always display 20 most frequent terms belonging to selected topics
    * click on a topic to select/color the topic
    * mouse over a topic/term to highlight the topic/term
(Note: the highest affinity term set and highest saliency term set may contain overlapping words)