Paper | arXiv | Code | Slides | Supplementary Results
Implementation of the Fast Dawid-Skene and Hybrid algorithms described in the paper:
Vaibhav B Sinha, Sukrut Rao, Vineeth N Balasubramanian. Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification. In Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM) at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2018, August 2018.
Implementations of the Dawid-Skene (Dawid and Skene, 1979) and Majority Voting algorithms are also provided.
These algorithms can be used to aggregate crowd-sourced labels to estimate the true labels. Given the labels of a data point from many annotators out of a set of classes, the algorithms output the most likely correct class for the data point.
The prerequisites are:
- Python 2.7 or 3.4-3.6
- pip
All other dependencies can be installed using pip, as
$ pip install -r requirements.txt
If tests are to be run, use instead,
$ pip install -r requirements-dev.txt
A description of the data format and the procedure to add a new dataset is given here. A toy dataset is also provided, and can be found here.
To run the program, use
$ python scripts/fast_dawid_skene.py [OPTIONS]
To view a list of available options along with descriptions, use
$ python scripts/fast_dawid_skene.py --help
To run on the toy dataset, with two annotators per question, using the FDS algorithm to obtain predictions, use
$ python scripts/fast_dawid_skene.py --dataset toy --k 2 --mode aggregate --algorithm FDS --print_result
To run using all available annotations for every question, using the FDS algorithm to obtain predictions, use
$ python scripts/fast_dawid_skene.py --dataset toy --mode aggregate --algorithm FDS --print_result
Tests can be run using pytest, as,
$ py.test
This code is provided under the MIT License.
Some parts of the code in this file are derived from this implementation, and the original license and copyright notice can be found at the top of the file.
If the Fast Dawid-Skene / Hybrid algorithms are useful for your research, please cite our paper [2] (References).
Parts of the code for the implementation of the algorithms use or derive from code in this implementation.
- A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. J. Royal Stat. Soc. Series C 28, 1 (1979), 20–28.
- Vaibhav B Sinha, Sukrut Rao, Vineeth N Balasubramanian. Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification. In Workshop on Issues of Sentiment Discovery and Opinion Mining (WISDOM) at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) 2018, August 2018.