Negation as a Source of Error in Machine Translation

This repository contains code of the EMNLP2020 (Findings) paper "It's not a Non-Issue: Negation as a Source of Error in Machine Translation". Download the paper from ACL anthology: https://www.aclweb.org/anthology/2020.findings-emnlp.345.pdf. The arxiv version is also available: https://arxiv.org/abs/2010.05432.
Authors: Md Mosharaf Hossain, Antonios Anastasopoulos, Eduardo Blanco and Alexis Palmer

Step 1: Download WMT-2018 and WMT-2019 corpora

To run the code, WMT-2018 and WMT-2019 corpora need to be downloaded.
At first, the project directory is needed to set as current directory.
run the script "download_wmt_files.sh" (execute permission of the script is required) inside the "data" directory.

cd data
sh download_wmt_files.sh

Step 2: Set-up environment

Python 3.5+ (recommended: Python 3.6)
Python packages: list of packages are provided in env-setup/requirements.txt file.

# Create virtual env (Assuming you have Python 3.5 or 3.6 installed in your machine) -> optional step
python3 -m venv your_location/negation-mt
source your_location/negation-mt/bin/activate

# Install required packages -> required step
pip install -r ./env-setup/requirements.txt
python -m spacy download en_core_web_sm
export HDF5_USE_FILE_LOCKING='FALSE'

Step 3: Map source, reference, and system output

At this step, we map source sentences with references and system outputs. We also put human evaluation score (z-score) for each system output.

python map.py

Mapping files are created in the below locations:
WMT-18: ./output/map_scores/2018/
WMT-19: ./output/map_scores/2019/
​

Step 4: Extract with and without negation

At this step, we run the cue detector to get two samples: sentences with negations and sentences without negation, for each participating system of a language direction.

python extract_negation.py

Output files will be created in below locations. "map_scores" directory contains with and without negation files. Whereas, "info" directory contains token-level cue information.
WMT-18: ./output/map_scores/2018/ (e.g.,fien_uedin.5631_ref_neg.txt (with neg file), fien_uedin.5631_ref_noneg.txt (without neg file))
WMT-19: ./output/map_scores/2019/
WMT-18: ./output/info/2018/
WMT-19: ./output/info/2019/

Step 5: Generate ranking files

This step produces system-level ranking based on z-scores for WMT-2018 and WMT-2019. The ranking file also contains other numbers such as BLUE and chrF++ scores. Please note that - for english to foreign language directions, with and without negation samples are created based on source language. On the other hand, for foreign to english directions, samples are created based on reference translation.

python ranking.py

​

Step 6: Results

The reported tables (1-4) can be generated by the below python script. The results might differ slightly as the cue detector detects slightly deferent negated sentences in each run (because of random initialization of the network weights in each run).

python results.py --table_no 1

Reference

@inproceedings{hossain-etal-2020-non,
    title = "It{'}s not a Non-Issue: Negation as a Source of Error in Machine Translation",
    author = "Hossain, Md Mosharaf  and
      Anastasopoulos, Antonios  and
      Blanco, Eduardo  and
      Palmer, Alexis",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.345",
    pages = "3869--3885",
    abstract = "As machine translation (MT) systems progress at a rapid pace, questions of their adequacy linger. In this study we focus on negation, a universal, core property of human language that significantly affects the semantics of an utterance. We investigate whether translating negation is an issue for modern MT systems using 17 translation directions as test bed. Through thorough analysis, we find that indeed the presence of negation can significantly impact downstream quality, in some cases resulting in quality reductions of more than 60{\%}. We also provide a linguistically motivated analysis that directly explains the majority of our findings. We release our annotations and code to replicate our analysis here: https://github.com/mosharafhossain/negation-mt.",
}