This repository contains supplementary material used in research working with MQM error annotations on the English-Croatian translation pair.
The annotations folder contains MQM error annotations of 100 sentences translated from English to Croatian by three different MT systems, annotated by two different annotators.
The scripts folder contains the original code to calculate inter-annotator agreement (IAA) and MQM annotation normalisation. Consult the readme for details on running the code. If you'd prefer to work with a more streamlined version of the code, Yuying Ye has been kind enough to do some refactoring. I suggest checking out the GitHub repository (it also has annotation data for English-Chinese!)
If you use any of this data or code, please cite the following paper(s):
@article{quantitative-2018,
title={Quantitative Fine-Grained Human Evaluation of Machine Translation Systems: a Case Study on English to Croatian},
author={Filip Klubi\v{c}ka and Antonio Toral and V\'{i}ctor M. S\'{a}nchez-Cartagena},
journal={Machine Translation},
year={2018}
}
You can download the paper here.
@article{fine-grained-2017,
title={Fine-grained human evaluation of neural versus phrase-based machine translation},
author={Filip Klubi\v{c}ka and Antonio Toral and V\'{i}ctor M. S\'{a}nchez-Cartagena},
journal={The Prague Bulletin of Mathematical Linguistics},
volume={108},
issue={1},
year={2017},
pages={121-132}
}
You can download the paper here.