We are releasing the data set used in the ACL paper
Training Neural Machine Translation To Apply Terminology Constraints. Georgiana Dinu, Prashant Mathur, Marcello Federico and Yaser Al-Onaizan. ACL 2019 (Short).
The folder contain 4 files in two folders and a script with a README.
The terminology files from IATE and Wiktionary data are in the form: {wikt,iate}.{NUM}.terminology.tsv where NUM represents the number of sentences.
Files containing 414 and 727 lines are the ones with perfect match on the target side. (Results in Table 2) Files containing 581 and 975 lines are the ones with approximate match on the target side. (Results in Table 4)
For example, terminology file `iate.414.terminology.tsv' contain 2N+2 columns where N is the number of terminology matches in a sentence.
0 21 interview Interview
1 51 prize Preisgeld
2 77 China China
3 80 trade Handel
4 86 side Seite
5 87 budgetary Haushaltsdefizit
6 88 side Seite solution Lösung
7 113 forced prostitution Zwangsprostitution
The first column is the sentence ID from the WMT 17 test set (containing 3000 lines) which can be downloaded from WMT 2017 website [1]. Each terminology match is represented in three columns: Source, Target and Target (BPE). You can ignore the third column of every terminology.
To get the news test 2017 with the terminology, run the following:
wget http://data.statmt.org/wmt17/translation-task/test.tgz && tar -xvf && cd test/
cat newstest2017-ende-src.en.sgm | grep "seg id" | perl -pe "s/<seg id=\"[0-9]*\">//g" | perl -pe "s/<\/seg>//g" > newstest2017.en
cat newstest2017-ende-ref.de.sgm | grep "seg id" | perl -pe "s/<seg id=\"[0-9]*\">//g" | perl -pe "s/<\/seg>//g" > newstest2017.de
for term_file in iate/iate.{414,581}.terminology.tsv
do
python print_lines.py -l=<(cut -f2 ${term_file}) < newstest2017.en > newstest2017-${term_file}.en
python print_lines.py -l=<(cut -f2 ${term_file}) < newstest2017.de > newstest2017-${term_file}.de
done
for term_file in wiktionary/wikt.{727,975}.terminology.tsv
do
python print_lines.py -l=<(cut -f2 ${term_file}) < newstest2017.en > newstest2017-${term_file}.en
python print_lines.py -l=<(cut -f2 ${term_file}) < newstest2017.de > newstest2017-${term_file}.de
done
Wiktionary is provided under CC-BY-SA 3.0 License. No copyright on IATE terminology. The script is released under Apache 2.0.