anlp-sciner-test-empty.conll

MedNLI	O
Is	O
Not	O
Immune	O
:	O
Natural	O
Language	O
Inference	O
Artifacts	O
in	O
the	O
Clinical	O
Domain	O

Crowdworker	O
-	O
constructed	O
natural	O
language	O
inference	O
(	O
NLI	O
)	O
datasets	O
have	O
been	O
found	O
to	O
contain	O
statistical	O
artifacts	O
associated	O
with	O
the	O
annotation	O
process	O
that	O
allow	O
hypothesis	O
-	O
only	O
classifiers	O
to	O
achieve	O
better	O
-	O
than	O
-	O
random	O
performance	O
(	O
Poliak	O
et	O
al	O
.	O
,	O
2018;Gururangan	O
et	O
al	O
.	O
,	O
2018;Tsuchiya	O
,	O
2018	O
)	O
.	O
We	O
investigate	O
whether	O
MedNLI	O
,	O
a	O
physician	O
-	O
annotated	O
dataset	O
with	O
premises	O
extracted	O
from	O
clinical	O
notes	O
,	O
contains	O
such	O
artifacts	O
(	O
Romanov	O
and	O
Shivade	O
,	O
2018	O
)	O
.	O
We	O
find	O
that	O
entailed	O
hypotheses	O
contain	O
generic	O
versions	O
of	O
specific	O
concepts	O
in	O
the	O
premise	O
,	O
as	O
well	O
as	O
modifiers	O
related	O
to	O
responsiveness	O
,	O
duration	O
,	O
and	O
probability	O
.	O
Neutral	O
hypotheses	O
feature	O
conditions	O
and	O
behaviors	O
that	O
co	O
-	O
occur	O
with	O
,	O
or	O
cause	O
,	O
the	O
condition(s	O
)	O
in	O
the	O
premise	O
.	O
Contradiction	O
hypotheses	O
feature	O
explicit	O
negation	O
of	O
the	O
premise	O
and	O
implicit	O
negation	O
via	O
assertion	O
of	O
good	O
health	O
.	O
Adversarial	O
filtering	O
demonstrates	O
that	O
performance	O
degrades	O
when	O
evaluated	O
on	O
the	O
difficult	O
subset	O
.	O
We	O
provide	O
partition	O
information	O
and	O
recommendations	O
for	O
alternative	O
dataset	O
construction	O
strategies	O
for	O
knowledge	O
-	O
intensive	O
domains	O
.	O

Introduction	O

In	O
the	O
clinical	O
domain	O
,	O
the	O
ability	O
to	O
conduct	O
natural	O
language	O
inference	O
(	O
NLI	O
)	O
on	O
unstructured	O
,	O
domainspecific	O
texts	O
such	O
as	O
patient	O
notes	O
,	O
pathology	O
reports	O
,	O
and	O
scientific	O
papers	O
,	O
plays	O
a	O
critical	O
role	O
in	O
the	O
development	O
of	O
predictive	O
models	O
and	O
clinical	O
decision	O
support	O
(	O
CDS	O
)	O
systems	O
.	O

Considerable	O
progress	O
in	O
domain	O
-	O
agnostic	O
NLI	O
has	O
been	O
facilitated	O
by	O
the	O
development	O
of	O
largescale	O
,	O
crowdworker	O
-	O
constructed	O
datasets	O
,	O
including	O
the	O
Stanford	O
Natural	O
Language	O
Inference	O
corpus	O
(	O
SNLI	O
)	O
,	O
and	O
the	O
Multi	O
-	O
Genre	O
Natural	O
Language	O
Inference	O
(	O
MultiNLI	O
)	O
corpus	O
(	O
Bowman	O
et	O
al	O
.	O
,	O
2015;Williams	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
MedNLI	O
is	O
a	O
similarlymotivated	O
,	O
healthcare	O
-	O
specific	O
dataset	O
created	O
by	O
a	O
small	O
team	O
of	O
physician	O
-	O
annotators	O
in	O
lieu	O
of	O
crowdworkers	O
,	O
due	O
to	O
the	O
extensive	O
domain	O
expertise	O
required	O
(	O
Romanov	O
and	O
Shivade	O
,	O
2018	O
)	O
.	O
Poliak	O
et	O
al	O
.	O
(	O
2018	O
)	O
,	O
Gururangan	O
et	O
al	O
.	O
(	O
2018	O
)	O
,	O
Tsuchiya	O
(	O
2018	O
)	O
,	O
andMcCoy	O
et	O
al	O
.	O
(	O
2019	O
)	O
empirically	O
demonstrate	O
that	O
SNLI	O
and	O
MultiNLI	O
contain	O
lexical	O
and	O
syntactic	O
annotation	O
artifacts	O
that	O
are	O
disproportionately	O
associated	O
with	O
specific	O
classes	O
,	O
allowing	O
a	O
hypothesis	O
-	O
only	O
classifier	O
to	O
significantly	O
outperform	O
a	O
majority	O
-	O
class	O
baseline	O
model	O
.	O
The	O
presence	O
of	O
such	O
artifacts	O
is	O
hypothesized	O
to	O
be	O
partially	O
attributable	O
to	O
the	O
priming	O
effect	O
of	O
the	O
example	O
hypotheses	O
provided	O
to	O
crowdworkers	O
at	O
annotation	O
-	O
time	O
.	O
Romanov	O
and	O
Shivade	O
(	O
2018	O
)	O
note	O
that	O
a	O
hypothesis	O
-	O
only	O
baseline	O
is	O
able	O
to	O
outperform	O
a	O
majority	O
class	O
baseline	O
in	O
MedNLI	O
,	O
but	O
they	O
do	O
not	O
identify	O
specific	O
artifacts	O
.	O

We	O
confirm	O
the	O
presence	O
of	O
annotation	O
artifacts	O
in	O
MedNLI	O
and	O
proceed	O
to	O
identify	O
their	O
lexical	O
and	O
semantic	O
characteristics	O
.	O
We	O
then	O
conduct	O
adversarial	O
filtering	O
to	O
partition	O
MedNLI	O
into	O
easy	O
and	O
difficult	O
subsets	O
(	O
Sakaguchi	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
We	O
find	O
that	O
performance	O
of	O
off	O
-	O
the	O
-	O
shelf	O
fastText	O
-	O
based	O
hypothesis	O
-	O
only	O
and	O
hypothesis	O
-	O
plus	O
-	O
premise	O
classifiers	O
is	O
lower	O
on	O
the	O
difficult	O
subset	O
than	O
on	O
the	O
full	O
and	O
easy	O
subsets	O
(	O
Joulin	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
We	O
provide	O
partition	O
information	O
for	O
downstream	O
use	O
,	O
and	O
conclude	O
by	O
advocating	O
alternative	O
dataset	O
construction	O
strategies	O
for	O
knowledge	O
-	O
intensive	O
domains	O
.	O
1	O

The	O
MedNLI	O
Dataset	O

MedNLI	O
is	O
domain	O
-	O
specific	O
evaluation	O
dataset	O
inspired	O
by	O
general	O
-	O
purpose	O
NLI	O
datasets	O
,	O
including	O
SNLI	O
and	O
MultiNLI	O
(	O
Romanov	O
and	O
Shivade	O
,	O
2018;Bowman	O
et	O
al	O
.	O
,	O
2015;Williams	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
Much	O
like	O
its	O
predecessors	O
,	O
MedNLI	O
consists	O
of	O
premisehypothesis	O
pairs	O
,	O
in	O
which	O
the	O
premises	O
are	O
drawn	O
1021	O
from	O
the	O
Past	O
Medical	O
History	O
sections	O
of	O
a	O
randomly	O
selected	O
subset	O
of	O
de	O
-	O
identified	O
clinical	O
notes	O
contained	O
in	O
MIMIC	O
-	O
III	O
(	O
Johnson	O
et	O
al	O
.	O
,	O
2016;Goldberger	O
et	O
al	O
.	O
,	O
2000	O
(	O
June	O
13	O
)	O
.	O
MIMIC	O
-	O
III	O
was	O
created	O
from	O
the	O
records	O
of	O
adult	O
and	O
neonatal	O
intensive	O
care	O
unit	O
(	O
ICU	O
)	O
patients	O
.	O
As	O
such	O
,	O
complex	O
and	O
clinically	O
severe	O
cases	O
are	O
disproportionately	O
represented	O
,	O
relative	O
to	O
their	O
frequency	O
of	O
occurrence	O
in	O
the	O
general	O
population	O
.	O

Physician	O
-	O
annotators	O
were	O
asked	O
to	O
write	O
a	O
definitely	O
true	O
,	O
maybe	O
true	O
,	O
and	O
definitely	O
false	O
set	O
of	O
hypotheses	O
for	O
each	O
premise	O
,	O
corresponding	O
to	O
entailment	O
,	O
neutral	O
and	O
contradiction	O
labels	O
,	O
respectively	O
.	O
The	O
resulting	O
dataset	O
has	O
cardinality	O
:	O
n	O
train	O
=	O
11232	O
;	O
n	O
dev	O
=	O
1395	O
;	O
n	O
test	O
=	O
1422	O
.	O

MedNLI	O
Contains	O
Artifacts	O

To	O
determine	O
whether	O
MedNLI	O
contains	O
annotation	O
artifacts	O
that	O
may	O
artificially	O
inflate	O
the	O
performance	O
of	O
models	O
trained	O
on	O
this	O
dataset	O
,	O
we	O
train	O
a	O
simple	O
,	O
premise	O
-	O
unaware	O
,	O
fastText	O
classifier	O
to	O
predict	O
the	O
label	O
of	O
each	O
premise	O
-	O
hypothesis	O
pair	O
,	O
and	O
compare	O
the	O
performance	O
of	O
this	O
classifier	O
to	O
a	O
majority	O
-	O
class	O
baseline	O
,	O
in	O
which	O
all	O
training	O
examples	O
are	O
mapped	O
to	O
the	O
most	O
commonly	O
occurring	O
class	O
label	O
(	O
Joulin	O
et	O
al	O
.	O
,	O
2016;Poliak	O
et	O
al	O
.	O
,	O
2018;Gururangan	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
Note	O
that	O
since	O
annotators	O
were	O
asked	O
to	O
create	O
an	O
entailed	O
,	O
contradictory	O
,	O
and	O
neutral	O
hypothesis	O
for	O
each	O
premise	O
,	O
MedNLI	O
is	O
class	O
-	O
balanced	O
.	O
Thus	O
,	O
in	O
this	O
setting	O
,	O
a	O
majority	O
class	O
baseline	O
is	O
equivalent	O
to	O
choosing	O
a	O
label	O
uniformly	O
at	O
random	O
for	O
each	O
training	O
example	O
.	O

The	O
micro	O
F1	O
-	O
score	O
achieved	O
by	O
the	O
fastText	O
classifier	O
significantly	O
exceeds	O
that	O
of	O
the	O
majority	O
class	O
baseline	O
,	O
confirming	O
the	O
findings	O
of	O
Romanov	O
and	O
Shivade	O
(	O
2018	O
)	O
,	O
who	O
report	O
a	O
micro	O
-	O
F1	O
score	O
of	O
61.9	O
but	O
do	O
not	O
identify	O
or	O
analyze	O
artifacts	O
:	O

Characteristics	O
of	O
Clinical	O
Artifacts	O

In	O
this	O
section	O
,	O
we	O
conduct	O
class	O
-	O
specific	O
lexical	O
analysis	O
to	O
identify	O
the	O
clinical	O
and	O
domainagnostic	O
characteristics	O
of	O
annotation	O
artifacts	O
associated	O
with	O
each	O
set	O
of	O
hypotheses	O
in	O
MedNLI	O
.	O

Preprocessing	O

We	O
cast	O
each	O
hypothesis	O
string	O
in	O
the	O
MedNLI	O
training	O
dataset	O
to	O
lowercase	O
.	O
We	O
then	O
use	O
a	O
scispaCy	O
model	O
pre	O
-	O
trained	O
on	O
the	O
en_core_sci_lg	O
corpus	O
for	O
tokenization	O
and	O
clinical	O
named	O
entity	O
recognition	O
(	O
CNER	O
)	O
(	O
Neumann	O
et	O
al	O
.	O
,	O
2019a	O
)	O
.	O
One	O
challenge	O
associated	O
with	O
clinical	O
text	O
,	O
and	O
scientific	O
text	O
more	O
generally	O
,	O
is	O
that	O
semantically	O
meaningful	O
entities	O
often	O
consist	O
of	O
spans	O
rather	O
than	O
single	O
tokens	O
.	O
To	O
mitigate	O
this	O
issue	O
during	O
lexical	O
analysis	O
,	O
we	O
map	O
each	O
multi	O
-	O
token	O
entity	O
to	O
a	O
single	O
-	O
token	O
representation	O
,	O
where	O
sub	O
-	O
tokens	O
are	O
separated	O
by	O
underscores	O
.	O

Lexical	O
Artifacts	O

Following	O
Gururangan	O
et	O
al	O
.	O
(	O
2018	O
)	O
,	O
to	O
identify	O
tokens	O
that	O
occur	O
disproportionately	O
in	O
hypotheses	O
associated	O
with	O
a	O
specific	O
class	O
,	O
we	O
compute	O
tokenclass	O
pointwise	O
mutual	O
information	O
(	O
PMI	O
)	O
with	O
add-50	O
smoothing	O
applied	O
to	O
raw	O
counts	O
,	O
and	O
a	O
filter	O
to	O
exclude	O
tokens	O
appearing	O
less	O
than	O
five	O
times	O
in	O
the	O
overall	O
training	O
dataset	O
.	O

Physician	O
-	O
Annotator	O
Heuristics	O

In	O
this	O
section	O
,	O
we	O
re	O
-	O
introduce	O
premises	O
to	O
our	O
analysis	O
to	O
evaluate	O
a	O
set	O
of	O
hypotheses	O
regarding	O
latent	O
,	O
class	O
-	O
specific	O
annotator	O
heuristics	O
.	O
If	O
annotators	O
do	O
employ	O
class	O
-	O
specific	O
heuristics	O
,	O
we	O
should	O
expect	O
the	O
semantic	O
contents	O
,	O
ϕ	O
,	O
of	O
a	O
given	O
hypothesis	O
,	O
h	O
∈	O
H	O
,	O
to	O
be	O
influenced	O
not	O
only	O
by	O
the	O
semantic	O
contents	O
of	O
its	O
associated	O
premise	O
,	O
p	O
∈	O
P	O
,	O
but	O
also	O
by	O
the	O
target	O
class	O
,	O
c	O
∈	O
C.	O

To	O
investigate	O
,	O
we	O
identify	O
a	O
set	O
of	O
heuristics	O
parameterized	O
by	O
ϕ(p	O
)	O
and	O
c	O
,	O
and	O
characterized	O
by	O
the	O
presence	O
of	O
a	O
set	O
of	O
heuristic	O
-	O
specific	O
Medical	O
Subject	O
Headings	O
(	O
MeSH	O
)	O
linked	O
entities	O
in	O
the	O
premise	O
and	O
hypothesis	O
of	O
each	O
heuristic	O
-	O
satisfying	O
example	O
.	O
These	O
heuristics	O
are	O
described	O
below	O
;	O
specific	O
MeSH	O
features	O
are	O
detailed	O
in	O
the	O
Appendix	O
.	O

Hypernym	O
Heuristic	O
This	O
heuristic	O
applies	O
when	O
the	O
premise	O
contains	O
clinical	O
condition(s	O
)	O
,	O
medication(s	O
)	O
,	O
finding(s	O
)	O
,	O
procedure(s	O
)	O
or	O
event(s	O
)	O
,	O
the	O
target	O
class	O
is	O
entailment	O
,	O
and	O
the	O
generated	O
hypothesis	O
contains	O
term(s	O
)	O
that	O
can	O
be	O
interpreted	O
as	O
super	O
-	O
types	O
for	O
a	O
subset	O
of	O
elements	O
in	O
the	O
premise	O
(	O
e.g.	O
,	O
clindamycin	O
<	O
:	O
antibiotic	O
)	O
.	O

Probable	O
Cause	O
Heuristic	O
This	O
heuristic	O
applies	O
when	O
the	O
premise	O
contains	O
clinical	O
condition(s	O
)	O
,	O
the	O
target	O
class	O
is	O
neutral	O
,	O
and	O
the	O
generated	O
hypothesis	O
provides	O
a	O
plausible	O
,	O
often	O
subjective	O
or	O
behavioral	O
,	O
causal	O
explanation	O
for	O
the	O
condition	O
,	O
finding	O
,	O
or	O
event	O
described	O
in	O
the	O
premise	O
(	O
e.g.	O
,	O
associating	O
altered	O
mental	O
status	O
with	O
drug	O
overdose	O
)	O
.	O

Everything	O
Is	O
Fine	O
Heuristic	O
This	O
heuristic	O
applies	O
when	O
the	O
premise	O
contains	O
condition(s	O
)	O
or	O
finding(s	O
)	O
,	O
the	O
target	O
class	O
is	O
contradiction	O
,	O
and	O
the	O
generated	O
hypothesis	O
negates	O
the	O
premise	O
or	O
asserts	O
unremarkable	O
finding(s	O
)	O
.	O
This	O
can	O
take	O
two	O
forms	O
:	O
repetition	O
of	O
premise	O
content	O
plus	O
negation	O
,	O
or	O
inclusion	O
of	O
modifiers	O
that	O
convey	O
good	O
health	O
.	O

Analysis	O
We	O
conduct	O
a	O
χ	O
2	O
test	O
for	O
each	O
heuristic	O
to	O
determine	O
whether	O
we	O
are	O
able	O
to	O
reject	O
the	O
null	O
hypothesis	O
that	O
pattern	O
-	O
satisfying	O
premisehypothesis	O
pairs	O
are	O
uniformly	O
distributed	O
over	O
classes	O
.	O
The	O
results	O
support	O
our	O
hypotheses	O
regarding	O
each	O
of	O
the	O
three	O
heuristics	O
.	O
Notably	O
,	O
the	O
percentage	O
of	O
heuristic	O
-	O
satisfying	O
pairs	O
accounted	O
for	O
by	O
the	O
top	O
class	O
is	O
lowest	O
for	O
the	O
HYPERNYM	O
hypothesis	O
,	O
which	O
we	O
attribute	O
to	O
the	O
high	O
degree	O
of	O
semantic	O
overlap	O
between	O
entailed	O
and	O
neutral	O
hypotheses	O
.	O

Adversarial	O
Filtering	O

To	O
mitigate	O
the	O
effect	O
of	O
clinical	O
annotation	O
artifacts	O
,	O
we	O
employ	O
AFLite	O
,	O
an	O
adversarial	O
filtering	O
algorithm	O
introduced	O
by	O
Sakaguchi	O
et	O
AFLite	O
requires	O
distributed	O
representations	O
of	O
the	O
full	O
dataset	O
as	O
input	O
,	O
and	O
proceeds	O
in	O
an	O
iterative	O
fashion	O
.	O
At	O
each	O
iteration	O
,	O
an	O
ensemble	O
of	O
n	O
linear	O
classifiers	O
are	O
trained	O
and	O
evaluated	O
on	O
different	O
random	O
subsets	O
of	O
the	O
data	O
.	O
A	O
score	O
is	O
then	O
computed	O
for	O
each	O
premise	O
-	O
hypothesis	O
instance	O
,	O
reflecting	O
the	O
number	O
of	O
times	O
the	O
instance	O
is	O
correctly	O
labeled	O
by	O
a	O
classifier	O
,	O
divided	O
by	O
the	O
number	O
of	O
times	O
the	O
instance	O
appears	O
in	O
any	O
classifier	O
's	O
evaluation	O
set	O
.	O
The	O
top	O
-	O
k	O
instances	O
with	O
scores	O
above	O
a	O
threshold	O
,	O
τ	O
,	O
are	O
filtered	O
out	O
and	O
added	O
to	O
the	O
easy	O
partition	O
;	O
the	O
remaining	O
instances	O
are	O
retained	O
.	O
This	O
process	O
continues	O
until	O
the	O
size	O
of	O
the	O
filtered	O
subset	O
is	O
<	O
k	O
,	O
or	O
the	O
number	O
of	O
retained	O
instances	O
is	O
<	O
m	O
;	O
retained	O
instances	O
constitute	O
the	O
difficult	O
partition	O
.	O

To	O
represent	O
the	O
full	O
dataset	O
,	O
we	O
use	O
fastText	O
MIMIC	O
-	O
III	O
embeddings	O
,	O
which	O
have	O
been	O
pretrained	O
on	O
deidentified	O
patient	O
notes	O
from	O
MIMIC	O
-	O
III	O
(	O
Romanov	O
and	O
Shivade	O
,	O
2018;Johnson	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
We	O
represent	O
each	O
example	O
as	O
the	O
average	O
of	O
its	O
component	O
token	O
vectors	O
.	O
We	O
proportionally	O
adjust	O
a	O
subset	O
of	O
the	O
hyperparameters	O
used	O
by	O
Sakaguchi	O
et	O
al	O
.	O
(	O
2020	O
)	O
to	O
account	O
for	O
the	O
fact	O
that	O
MedNLI	O
contains	O
far	O
fewer	O
examples	O
than	O
WINOGRANDE	O
2	O
:	O
specifically	O
,	O
we	O
set	O
the	O
training	O
size	O
for	O
each	O
ensemble	O
,	O
m	O
,	O
to	O
5620	O
,	O
which	O
represents	O
≈	O
2	O
5	O
of	O
the	O
MedNLI	O
combined	O
dataset	O
.	O
The	O
remaining	O
hyperparameters	O
are	O
unchanged	O
:	O
the	O
ensemble	O
consists	O
of	O
n	O
=	O
64	O
logistic	O
regression	O
models	O
,	O
the	O
filtering	O
cutoff	O
,	O
k	O
=	O
500	O
,	O
and	O
the	O
filtering	O
threshold	O
τ	O
=	O
0.75	O
.	O

We	O
apply	O
AFLite	O
to	O
two	O
different	O
versions	O
of	O
MedNLI	O
:	O
(	O
1	O
)	O
X	O
h	O
,	O
m	O
:	O
hypothesis	O
-	O
only	O
,	O
multi	O
-	O
token	O
entities	O
merged	O
,	O
and	O
(	O
2	O
)	O
X	O
ph	O
,	O
m	O
:	O
premise	O
and	O
hypothesis	O
concatenated	O
,	O
multi	O
-	O
token	O
entities	O
merged	O
.	O
AFLIte	O
maps	O
each	O
version	O
to	O
an	O
easy	O
and	O
difficult	O
partition	O
,	O
which	O
can	O
in	O
turn	O
be	O
split	O
into	O
training	O
,	O
dev	O
,	O
and	O
test	O
subsets	O
.	O
We	O
report	O
results	O
for	O
the	O
fastText	O
classifier	O
trained	O
on	O
the	O
original	O
,	O
hypothesis	O
-	O
only	O
(	O
hypothesis	O
+	O
premise	O
)	O
MedNLI	O
training	O
set	O
,	O
and	O
evaluated	O
on	O
the	O
full	O
,	O
easy	O
and	O
difficult	O
dev	O
and	O
test	O
subsets	O
of	O
X	O
h	O
,	O
m	O
(	O
X	O
ph	O
,	O
m	O
)	O
,	O
and	O
observe	O
that	O
performance	O
decreases	O
on	O
the	O
difficult	O
partition	O
:	O

Discussion	O

MedNLI	O
is	O
Not	O
Immune	O
from	O
Artifacts	O

In	O
this	O
paper	O
,	O
we	O
demonstrate	O
that	O
MedNLI	O
suffers	O
from	O
the	O
same	O
challenge	O
associated	O
with	O
annotation	O
artifacts	O
that	O
its	O
domain	O
-	O
agnostic	O
predecessors	O
have	O
encountered	O
:	O
namely	O
,	O
NLI	O
models	O
trained	O
on	O
{	O
Med	O
,	O
S	O
,	O
Multi}NLI	O
can	O
perform	O
well	O
even	O
without	O
access	O
to	O
the	O
training	O
examples	O
'	O
premises	O
,	O
indicating	O
that	O
they	O
often	O
exploit	O
shallow	O
heuristics	O
,	O
with	O
negative	O
implications	O
for	O
out	O
-	O
of	O
-	O
sample	O
generalization	O
.	O
Interestingly	O
,	O
many	O
of	O
the	O
high	O
-	O
level	O
lexical	O
characteristics	O
identified	O
in	O
MedNLI	O
can	O
be	O
considered	O
domain	O
-	O
specific	O
variants	O
of	O
the	O
more	O
generic	O
,	O
classspecific	O
patterns	O
identified	O
in	O
SNLI	O
.	O
This	O
observation	O
suggests	O
that	O
a	O
set	O
of	O
abstract	O
design	O
patterns	O
for	O
inference	O
example	O
generation	O
exists	O
across	O
domains	O
,	O
and	O
may	O
be	O
reinforced	O
by	O
the	O
prompts	O
provided	O
to	O
annotators	O
.	O
Creative	O
or	O
randomized	O
priming	O
,	O
such	O
as	O
Sakaguchi	O
et	O
al	O
.	O
(	O
2020	O
)	O
's	O
use	O
of	O
anchor	O
words	O
from	O
WikiHow	O
articles	O
,	O
may	O
help	O
to	O
decrease	O
reliance	O
on	O
such	O
design	O
patterns	O
,	O
but	O
it	O
appears	O
unlikely	O
that	O
they	O
can	O
be	O
systematically	O
sidestepped	O
without	O
introducing	O
new	O
,	O
"	O
corrective	O
"	O
artifacts	O
.	O

A	O
Prescription	O
for	O
Dataset	O
Construction	O

To	O
mitigate	O
the	O
risk	O
of	O
performance	O
overestimation	O
associated	O
with	O
annotation	O
artifacts	O
,	O
Zellers	O
et	O
al	O
.	O
(	O
2019	O
)	O
advocate	O
adversarial	O
dataset	O
construction	O
,	O
such	O
that	O
benchmarks	O
will	O
co	O
-	O
evolve	O
with	O
language	O
models	O
.	O
This	O
may	O
be	O
difficult	O
to	O
scale	O
in	O
knowledge	O
-	O
intensive	O
domains	O
,	O
as	O
expert	O
validation	O
of	O
adversarially	O
generated	O
benchmarks	O
is	O
typically	O
required	O
.	O
Additionally	O
,	O
in	O
high	O
-	O
stakes	O
domains	O
such	O
as	O
medicine	O
,	O
information	O
-	O
rich	O
inferences	O
should	O
be	O
preferred	O
over	O
correct	O
but	O
trivial	O
inferences	O
that	O
time	O
-	O
constrained	O
expert	O
annotators	O
may	O
be	O
rationally	O
incentivized	O
to	O
produce	O
,	O
because	O
entropy	O
-	O
reducing	O
inferences	O
are	O
more	O
useful	O
for	O
downstream	O
tasks	O
.	O

We	O
advocate	O
the	O
adoption	O
of	O
a	O
mechanism	O
design	O
perspective	O
,	O
so	O
as	O
to	O
develop	O
modified	O
annotation	O
tasks	O
that	O
reduce	O
the	O
cognitive	O
load	O
placed	O
on	O
expert	O
annotators	O
while	O
incentivizing	O
the	O
production	O
of	O
domain	O
-	O
specific	O
NLI	O
datasets	O
with	O
high	O
downstream	O
utility	O
(	O
Ho	O
et	O
al	O
.	O
,	O
2015;Liu	O
and	O
Chen	O
,	O
2017	O
)	O
.	O
An	O
additional	O
option	O
is	O
to	O
narrow	O
the	O
generative	O
scope	O
by	O
defining	O
a	O
set	O
of	O
inferences	O
deemed	O
to	O
be	O
useful	O
for	O
a	O
specific	O
task	O
.	O
Annotators	O
can	O
then	O
map	O
(	O
premise	O
,	O
relation	O
)	O
tuples	O
to	O
relation	O
-	O
satisfying	O
,	O
potentially	O
fuzzy	O
subsets	O
of	O
this	O
pool	O
of	O
useful	O
inferences	O
,	O
or	O
return	O
partial	O
functions	O
when	O
more	O
information	O
is	O
needed	O
.	O

Ethical	O
Considerations	O

When	O
working	O
with	O
clinical	O
data	O
,	O
two	O
key	O
ethical	O
objectives	O
include	O
:	O
(	O
1	O
)	O
the	O
preservation	O
of	O
pa	O
-	O
tient	O
privacy	O
,	O
and	O
(	O
2	O
)	O
the	O
development	O
of	O
language	O
and	O
predictive	O
models	O
that	O
benefit	O
patients	O
and	O
providers	O
to	O
the	O
extent	O
possible	O
,	O
without	O
causing	O
undue	O
harm	O
.	O
With	O
respect	O
to	O
the	O
former	O
,	O
MedNLI	O
's	O
premises	O
are	O
sampled	O
from	O
de	O
-	O
identified	O
clinical	O
notes	O
contained	O
in	O
MIMIC	O
-	O
III	O
(	O
Goldberger	O
et	O
al	O
.	O
,	O
2000	O
(	O
June	O
13;Johnson	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
and	O
the	O
hypotheses	O
generated	O
by	O
annotators	O
do	O
not	O
refer	O
to	O
specific	O
patients	O
,	O
providers	O
,	O
or	O
locations	O
by	O
name	O
.	O
MedNLI	O
requires	O
users	O
to	O
complete	O
Health	O
Insurance	O
Portability	O
and	O
Accountability	O
Act	O
(	O
HIPAA	O
)	O
training	O
and	O
sign	O
a	O
data	O
use	O
agreement	O
prior	O
to	O
being	O
granted	O
access	O
,	O
which	O
we	O
have	O
complied	O
with	O
.	O

Per	O
MedNLI	O
's	O
data	O
use	O
agreement	O
requirements	O
,	O
we	O
do	O
not	O
attempt	O
to	O
identify	O
any	O
patient	O
,	O
provider	O
,	O
or	O
institution	O
mentioned	O
in	O
the	O
de	O
-	O
identified	O
corpus	O
.	O
Additionally	O
,	O
while	O
we	O
provide	O
AFLite	O
easy	O
and	O
difficult	O
partition	O
information	O
for	O
community	O
use	O
in	O
the	O
form	O
of	O
split	O
-	O
example	O
ids	O
and	O
a	O
checksum	O
,	O
we	O
do	O
not	O
share	O
the	O
premise	O
or	O
hypothesis	O
text	O
associated	O
with	O
any	O
example	O
.	O
Interested	O
readers	O
are	O
encouraged	O
to	O
complete	O
the	O
necessary	O
training	O
and	O
obtain	O
credentials	O
so	O
that	O
they	O
can	O
access	O
the	O
complete	O
dataset	O
(	O
Romanov	O
and	O
Shivade	O
,	O
2018;Goldberger	O
et	O
al	O
.	O
,	O
2000	O
(	O
June	O
13	O
)	O
.	O

With	O
respect	O
to	O
benefiting	O
patients	O
,	O
the	O
discussion	O
of	O
natural	O
language	O
artifacts	O
we	O
have	O
presented	O
is	O
intended	O
to	O
encourage	O
clinical	O
researchers	O
who	O
rely	O
on	O
(	O
or	O
construct	O
)	O
expert	O
-	O
annotated	O
clinical	O
corpora	O
to	O
train	O
domain	O
-	O
specific	O
language	O
models	O
,	O
or	O
consume	O
such	O
models	O
to	O
perform	O
downstream	O
tasks	O
,	O
to	O
be	O
aware	O
of	O
the	O
presence	O
of	O
annotation	O
artifacts	O
,	O
and	O
adjust	O
their	O
assessments	O
of	O
model	O
performance	O
accordingly	O
.	O
It	O
is	O
our	O
hope	O
that	O
these	O
findings	O
can	O
be	O
used	O
to	O
inform	O
error	O
analysis	O
and	O
improve	O
predictive	O
models	O
that	O
inform	O
patient	O
care	O
.	O

A	O
Appendix	O

A.1	O
Hypothesis	O
-	O
only	O
Baseline	O
Analysis	O

To	O
conduct	O
the	O
analysis	O
presented	O
in	O
Section	O
3	O
,	O
we	O
take	O
the	O
MedNLI	O
training	O
dataset	O
as	O
input	O
,	O
and	O
exclude	O
the	O
premise	O
text	O
for	O
each	O
training	O
example	O
.	O
We	O
cast	O
the	O
text	O
of	O
each	O
training	O
hypothesis	O
to	O
lowercase	O
,	O
but	O
do	O
not	O
perform	O
any	O
additional	O
preprocessing	O
.	O
We	O
use	O
an	O
off	O
-	O
the	O
-	O
shelf	O
fastText	O
classifier	O
,	O
with	O
all	O
model	O
hyperparameters	O
set	O
to	O
their	O
default	O
values	O
with	O
the	O
exception	O
of	O
wordNgrams	O
,	O
which	O
we	O
set	O
equal	O
to	O
2	O
to	O
allow	O
the	O
model	O
to	O
use	O
bigrams	O
in	O
addition	O
to	O
unigrams	O
(	O
Joulin	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
We	O
evaluate	O
the	O
trained	O
classifier	O
on	O
the	O
hypotheses	O
contained	O
in	O
the	O
MedNLI	O
dev	O
and	O
test	O
datasets	O
,	O
and	O
report	O
results	O
for	O
each	O
split	O
.	O

A.2	O
Lexical	O
Artifact	O
Analysis	O

To	O
perform	O
the	O
analysis	O
presented	O
in	O
Section	O
4	O
,	O
we	O
cast	O
each	O
hypothesis	O
string	O
in	O
the	O
MedNLI	O
training	O
dataset	O
to	O
lowercase	O
.	O
We	O
then	O
use	O
a	O
scispaCy	O
model	O
pre	O
-	O
trained	O
on	O
the	O
en_core_sci_lg	O
corpus	O
for	O
tokenization	O
and	O
clinical	O
named	O
entity	O
recognition	O
(	O
CNER	O
)	O
(	O
Neumann	O
et	O
al	O
.	O
,	O
2019a	O
)	O
.	O
Next	O
,	O
we	O
merge	O
multi	O
-	O
token	O
entities	O
,	O
using	O
underscores	O
as	O
delimiters	O
-	O
e.g.	O
,	O
"	O
brain	O
injury	O
"	O
→	O
"	O
brain_injury	O
"	O
.	O

When	O
computing	O
token	O
-	O
class	O
pointwise	O
mutual	O
information	O
(	O
PMI	O
)	O
,	O
we	O
exclude	O
tokens	O
that	O
appear	O
less	O
than	O
five	O
times	O
in	O
the	O
overall	O
training	O
dataset	O
's	O
hypotheses	O
.	O
Then	O
,	O
following	O
Gururangan	O
et	O
al	O
.	O
(	O
2018	O
)	O
,	O
who	O
apply	O
add-100	O
smoothing	O
to	O
raw	O
counts	O
to	O
highlight	O
particularly	O
discriminative	O
token	O
-	O
class	O
co	O
-	O
occurrence	O
patterns	O
,	O
we	O
apply	O
add-50	O
smoothing	O
to	O
raw	O
counts	O
.	O
Our	O
approach	O
is	O
similarly	O
motivated	O
;	O
our	O
choice	O
of	O
50	O
reflects	O
the	O
smaller	O
state	O
space	O
associated	O
with	O
a	O
focus	O
on	O
the	O
clinical	O
domain	O
.	O

A.3	O
Semantic	O
Analysis	O
of	O
Heuristics	O

To	O
perform	O
the	O
statistical	O
analysis	O
presented	O
in	O
Section	O
5	O
,	O
we	O
take	O
the	O
premise	O
-	O
hypothesis	O
pairs	O
from	O
the	O
MedNLI	O
training	O
,	O
dev	O
,	O
and	O
test	O
splits	O
,	O
and	O
combine	O
them	O
to	O
produce	O
a	O
single	O
corpus	O
.	O
We	O
use	O
a	O
scispaCy	O
model	O
pre	O
-	O
trained	O
on	O
the	O
en_core_sci_lg	O
corpus	O
for	O
tokenization	O
and	O
entity	O
linking	O
(	O
Neumann	O
et	O
al	O
.	O
,	O
2019b	O
)	O
,	O
and	O
link	O
against	O
the	O
Medical	O
Subject	O
Headings	O
(	O
MeSH	O
)	O
knowledge	O
base	O
.	O
We	O
take	O
the	O
top	O
-	O
ranked	O
knowledge	O
base	O
entry	O
for	O
each	O
linked	O
entity	O
.	O
Linking	O
against	O
MeSH	O
provides	O
a	O
unique	O
concept	O
i	O
d	O
,	O
canonical	O
name	O
,	O
alias(es	O
)	O
,	O
a	O
definition	O
,	O
and	O
one	O
or	O
more	O
MeSH	O
tree	O
numbers	O
for	O
each	O
recovered	O
entity	O
.	O
Tree	O
numbers	O
convey	O
semantic	O
type	O
information	O
by	O
embedding	O
each	O
concept	O
into	O
the	O
broader	O
MeSH	O
hierarchy	O
3	O
.	O
We	O
operationalize	O
each	O
of	O
our	O
heuristics	O
with	O
a	O
set	O
of	O
MeSH	O
-	O
informed	O
semantic	O
properties	O
,	O
which	O
are	O
defined	O
as	O
follows	O
:	O

1	O
.	O
Hypernym	O
Heuristic	O
:	O
a	O
premise	O
-	O
hypothesis	O
pair	O
satisfies	O
this	O
heuristic	O
if	O
specific	O
clinical	O
concept(s	O
)	O
appearing	O
in	O
the	O
premise	O
appear	O
in	O
a	O
more	O
general	O
form	O
in	O
the	O
hypothesis	O
.	O
Formally	O
:	O
{	O
(	O
p	O
,	O
h)|ϕ(p	O
)	O
ϕ(h	O
)	O
}	O
.	O
MeSH	O
tree	O
numbers	O
are	O
organized	O
hierarchically	O
,	O
and	O
increase	O
in	O
length	O
with	O
specificity	O
.	O
Thus	O
,	O
when	O
a	O
premise	O
entity	O
and	O
hypothesis	O
entity	O
are	O
leftaligned	O
,	O
the	O
hypothesis	O
entity	O
is	O
a	O
hypernym	O
for	O
the	O
premise	O
entity	O
if	O
the	O
hypothesis	O
entity	O
is	O
a	O
substring	O
of	O
the	O
premise	O
entity	O
.	O
To	O
provide	O
a	O
concrete	O
example	O
:	O
diabetes	O
mellitus	O
is	O
an	O
endocrine	O
system	O
disease	O
;	O
the	O
associated	O
MeSH	O
tree	O
numbers	O
are	O
C19.246	O
and	O
C19	O
,	O
respectively	O
.	O

2	O
.	O
Probable	O
Cause	O
Heuristic	O
:	O
a	O
premisehypothesis	O
pair	O
satisfies	O
this	O
heuristic	O
if	O
:	O
(	O
1	O
)	O
the	O
premise	O
contains	O
one	O
or	O
more	O
MeSH	O
entities	O
belonging	O
to	O
high	O
-	O
level	O
categories	O
C	O
(	O
diseases	O
)	O
,	O
D	O
(	O
chemicals	O
and	O
drugs	O
)	O
,	O
E	O
(	O
analytical	O
,	O
diagnostic	O
and	O
therapeutic	O
techniques	O
,	O
and	O
equipment	O
)	O
or	O
F	O
(	O
psychiatry	O
and	O
psychology	O
)	O
;	O
and	O
(	O
2	O
)	O
the	O
hypothesis	O
contains	O
one	O
or	O
more	O
MeSH	O
entities	O
that	O
can	O
be	O
interpreted	O
as	O
providing	O
a	O
plausible	O
causal	O
or	O
behavioral	O
explanation	O
for	O
the	O
condition	O
,	O
finding	O
,	O
or	O
event	O
described	O
in	O
the	O
premise	O
(	O
e.g.	O
,	O
smoking	O
,	O
substance	O
-	O
related	O
disorders	O
,	O
mental	O
disorders	O
,	O
alcoholism	O
,	O
homelessness	O
,	O
obesity	O
)	O
.	O

3	O
.	O
Everything	O
Is	O
Fine	O
Heuristic	O
:	O
a	O
premisehypothesis	O
pair	O
satisfies	O
this	O
heuristic	O
if	O
the	O
hypothesis	O
contains	O
one	O
or	O
more	O
of	O
the	O
same	O
MeSH	O
entities	O
as	O
the	O
premise	O
(	O
excluding	O
the	O
patient	O
entity	O
,	O
which	O
appears	O
in	O
almost	O
all	O
notes	O
)	O
and	O
also	O
contains	O
:	O
(	O
1	O
)	O
a	O
negation	O
word	O
or	O
phrase	O
(	O
e.g.	O
,	O
does	O
not	O
have	O
,	O
no	O
finding	O
,	O
no	O
,	O
denies	O
)	O
;	O
or	O
(	O
2	O
)	O
a	O
word	O
or	O
phrase	O
that	O
affirms	O
the	O
patient	O
's	O
health	O
(	O
e.g.	O
,	O
normal	O
,	O
healthy	O
,	O
discharged	O
)	O
.	O

For	O
each	O
heuristic	O
,	O
we	O
subset	O
the	O
complete	O
dataset	O
to	O
find	O
pattern	O
-	O
satisfying	O
premise	O
-	O
heuristic	O
pairs	O
.	O
We	O
use	O
this	O
subset	O
when	O
performing	O
the	O
χ	O
2	O
tests	O
.	O

Acknowledgments	O

We	O
thank	O
the	O
four	O
anonymous	O
reviewers	O
whose	O
feedback	O
and	O
suggestions	O
helped	O
improve	O
this	O
manuscript	O
.	O
The	O
first	O
author	O
was	O
supported	O
by	O
the	O
National	O
Institute	O
of	O
Standards	O
and	O
Technology	O
's	O
(	O
NIST	O
)	O
Professional	O
Research	O
Experience	O
Program	O
(	O
PREP	O
)	O
.	O
This	O
research	O
was	O
also	O
supported	O
by	O
the	O
DARPA	O
KAIROS	O
program	O
.	O
The	O
views	O
and	O
conclusions	O
contained	O
in	O
this	O
publication	O
are	O
those	O
of	O
the	O
authors	O
and	O
should	O
not	O
be	O
interpreted	O
as	O
representing	O
official	O
policies	O
or	O
endorsements	O
of	O
NIST	O
,	O
DARPA	O
,	O
or	O
the	O
U.S.	O
Government	O
.	O

A.4	O
Adversarial	O
Filtering	O

When	O
implementing	O
AFLite	O
,	O
we	O
follow	O
Sakaguchi	O
et	O
al	O
.	O
(	O
2020	O
)	O
.	O
We	O
use	O
a	O
smaller	O
training	O
set	O
size	O
of	O
m	O
=	O
5620	O
,	O
but	O
keep	O
the	O
remaining	O
hyperparameters	O
unchanged	O
,	O
such	O
that	O
the	O
ensemble	O
consists	O
of	O
n	O
=	O
64	O
logistic	O
regression	O
models	O
,	O
the	O
filtering	O
cutoff	O
,	O
k	O
=	O
500	O
,	O
and	O
the	O
filtering	O
threshold	O
τ	O
=	O
0.75	O
.	O

Question	O
Generation	O
for	O
Adaptive	O
Education	O

Intelligent	O
and	O
adaptive	O
online	O
education	O
systems	O
aim	O
to	O
make	O
high	O
-	O
quality	O
education	O
available	O
for	O
a	O
diverse	O
range	O
of	O
students	O
.	O
However	O
,	O
existing	O
systems	O
usually	O
depend	O
on	O
a	O
pool	O
of	O
hand	O
-	O
made	O
questions	O
,	O
limiting	O
how	O
finegrained	O
and	O
open	O
-	O
ended	O
they	O
can	O
be	O
in	O
adapting	O
to	O
individual	O
students	O
.	O
We	O
explore	O
targeted	O
question	O
generation	O
as	O
a	O
controllable	O
sequence	O
generation	O
task	O
.	O
We	O
first	O
show	O
how	O
to	O
fine	O
-	O
tune	O
pre	O
-	O
trained	O
language	O
models	O
for	O
deep	O
knowledge	O
tracing	O
(	O
LM	O
-	O
KT	O
)	O
.	O
This	O
model	O
accurately	O
predicts	O
the	O
probability	O
of	O
a	O
student	O
answering	O
a	O
question	O
correctly	O
,	O
and	O
generalizes	O
to	O
questions	O
not	O
seen	O
in	O
training	O
.	O
We	O
then	O
use	O
LM	O
-	O
KT	O
to	O
specify	O
the	O
objective	O
and	O
data	O
for	O
training	O
a	O
model	O
to	O
generate	O
questions	O
conditioned	O
on	O
the	O
student	O
and	O
target	O
difficulty	O
.	O
Our	O
results	O
show	O
we	O
succeed	O
at	O
generating	O
novel	O
,	O
wellcalibrated	O
language	O
translation	O
questions	O
for	O
second	O
language	O
learners	O
from	O
a	O
real	O
online	O
education	O
platform	O
.	O

Introduction	O

Online	O
education	O
platforms	O
can	O
increase	O
the	O
accessibility	O
of	O
educational	O
resources	O
around	O
the	O
world	O
.	O
However	O
,	O
achieving	O
equitable	O
outcomes	O
across	O
diverse	O
learning	O
needs	O
benefits	O
from	O
systems	O
that	O
are	O
adaptive	O
and	O
individualized	O
to	O
each	O
student	O
(	O
Doroudi	O
and	O
Brunskill	O
,	O
2019	O
)	O
.	O
Traditionally	O
,	O
adaptive	O
education	O
methods	O
involve	O
planning	O
over	O
a	O
pool	O
of	O
pre	O
-	O
made	O
questions	O
(	O
Atkinson	O
,	O
1972;Hunziker	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
These	O
are	O
naturally	O
limited	O
by	O
the	O
diversity	O
and	O
coverage	O
of	O
the	O
pool	O
,	O
as	O
well	O
as	O
the	O
scaling	O
capacity	O
of	O
curriculum	O
planning	O
algorithms	O
.	O
Recent	O
approaches	O
,	O
such	O
as	O
procedural	O
generation	O
for	O
personalized	O
programming	O
games	O
(	O
Valls	O
-	O
Vargas	O
et	O
al	O
.	O
,	O
2017	O
)	O
,	O
are	O
limited	O
to	O
well	O
-	O
specified	O
small	O
domains	O
.	O
We	O
address	O
these	O
limitations	O
by	O
leveraging	O
recent	O
success	O
in	O
deep	O
generative	O
models	O
,	O
in	O
particular	O
language	O
models	O
(	O
LMs	O
)	O
.	O

Many	O
educational	O
activities	O
involve	O
sequential	O
data	O
,	O
such	O
as	O
language	O
translation	O
,	O
reading	O
compre-	O

<	O
Y	O
>	O

Figure	O
1	O
:	O
Example	O
input	O
and	O
outputs	O
for	O
our	O
LM	O
-	O
based	O
knowledge	O
tracing	O
model	O
(	O
middle	O
)	O
and	O
question	O
generation	O
model	O
(	O
bottom	O
)	O
for	O
an	O
online	O
reverse	O
language	O
translation	O
task	O
(	O
top	O
)	O
.	O
A	O
question	O
in	O
this	O
task	O
consists	O
of	O
a	O
target	O
phrase	O
for	O
the	O
student	O
,	O
in	O
this	O
case	O
a	O
Spanish	O
learner	O
,	O
to	O
translate	O
(	O
e.g.	O
"	O
the	O
woman	O
"	O
)	O
.	O

hension	O
,	O
algebra	O
,	O
and	O
deductive	O
logic	O
.	O
Meanwhile	O
,	O
pre	O
-	O
trained	O
LMs	O
can	O
effectively	O
handle	O
sequences	O
from	O
a	O
wide	O
range	O
of	O
modalities	O
(	O
Madani	O
et	O
al	O
.	O
,	O
2020;Polu	O
and	O
Sutskever	O
,	O
2020	O
)	O
.	O
In	O
this	O
work	O
,	O
we	O
focus	O
on	O
natural	O
language	O
sequences	O
,	O
where	O
recent	O
progress	O
in	O
language	O
modeling	O
has	O
shown	O
great	O
success	O
at	O
capturing	O
abstract	O
properties	O
of	O
language	O
(	O
Hewitt	O
and	O
Manning	O
,	O
2019;Liu	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Specifically	O
,	O
we	O
show	O
how	O
pre	O
-	O
trained	O
LMs	O
can	O
be	O
easily	O
leveraged	O
to	O
adaptively	O
generate	O
questions	O
for	O
a	O
given	O
student	O
and	O
target	O
difficulty	O
in	O
a	O
reverse	O
translation	O
task	O
,	O
using	O
difficulty	O
at	O
answering	O
questions	O
as	O
a	O
proxy	O
for	O
more	O
complex	O
future	O
learning	O
objectives	O
.	O

We	O
introduce	O
an	O
LM	O
-	O
based	O
knowledge	O
tracing	O
model	O
(	O
LM	O
-	O
KT	O
)	O
to	O
predict	O
students	O
'	O
difficulty	O
on	O
novel	O
questions	O
(	O
e.g.	O
target	O
phrases	O
to	O
translate	O
)	O
.	O
We	O
show	O
that	O
LM	O
-	O
KT	O
is	O
well	O
-	O
calibrated	O
,	O
allowing	O
us	O
to	O
pose	O
the	O
learning	O
problem	O
for	O
the	O
question	O
generator	O
:	O
given	O
a	O
student	O
state	O
,	O
generate	O
a	O
question	O
that	O
will	O
achieve	O
a	O
target	O
difficulty	O
,	O
according	O
to	O
LM	O
-	O
KT	O
.	O
We	O
evaluate	O
both	O
LM	O
-	O
KT	O
and	O
question	O
generation	O
models	O
on	O
real	O
users	O
and	O
responses	O
from	O
Duolingo	O
1	O
,	O
a	O
popular	O
online	O
second	O
-	O
language	O
learning	O
platform	O
.	O

Background	O
&	O
Related	O
Works	O

There	O
exists	O
a	O
rich	O
body	O
of	O
work	O
on	O
precisely	O
modeling	O
student	O
"	O
ability	O
"	O
and	O
learning	O
.	O
For	O
example	O
,	O
Item	O
Response	O
Theory	O
(	O
IRT	O
)	O
seeks	O
to	O
model	O
individual	O
student	O
ability	O
based	O
on	O
their	O
responses	O
to	O
different	O
questions	O
,	O
creating	O
a	O
strong	O
factorization	O
between	O
students	O
and	O
test	O
items	O
(	O
Lord	O
,	O
1980;Hambelton	O
and	O
Jodoin	O
,	O
2003	O
)	O
.	O
Meanwhile	O
,	O
Computer	O
Adaptive	O
Testing	O
(	O
CAT	O
)	O
techniques	O
are	O
used	O
to	O
determine	O
a	O
fixed	O
student	O
ability	O
as	O
quickly	O
as	O
possible	O
by	O
selecting	O
test	O
items	O
based	O
on	O
information	O
utility	O
(	O
Weiss	O
and	O
Kingsbury	O
,	O
1984;Thissen	O
and	O
Mislevy	O
,	O
2000;Settles	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
However	O
,	O
these	O
methods	O
,	O
which	O
have	O
been	O
used	O
to	O
develop	O
efficient	O
standardized	O
tests	O
,	O
do	O
not	O
necessarily	O
optimize	O
a	O
student	O
's	O
learning	O
experience	O
(	O
Mu	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
We	O
instead	O
focus	O
on	O
tracking	O
each	O
student	O
's	O
evolving	O
knowledge	O
,	O
choosing	O
questions	O
to	O
target	O
difficulty	O
.	O

Knowledge	O
Tracing	O
(	O
KT	O
)	O
seeks	O
to	O
model	O
a	O
student	O
's	O
knowledge	O
state	O
from	O
their	O
answer	O
history	O
in	O
order	O
to	O
help	O
individualize	O
exercise	O
sequences	O
(	O
Corbett	O
and	O
Anderson	O
,	O
1995	O
)	O
.	O
This	O
draws	O
inspiration	O
from	O
traditional	O
education	O
curriculum	O
practices	O
,	O
such	O
as	O
distributed	O
spacing	O
of	O
vocabulary	O
(	O
Bloom	O
and	O
Shuell	O
,	O
1981	O
)	O
and	O
mixed	O
review	O
in	O
mathematics	O
(	O
Rohrer	O
,	O
2009	O
)	O
.	O
To	O
address	O
simplifying	O
assumptions	O
in	O
earlier	O
KT	O
approaches	O
,	O
such	O
as	O
discrete	O
knowledge	O
representations	O
,	O
Piech	O
et	O
al	O
.	O
(	O
2015	O
)	O
introduced	O
Deep	O
Knowledge	O
Tracing	O
(	O
DKT	O
)	O
,	O
which	O
uses	O
RNNs	O
to	O
enable	O
more	O
complex	O
knowledge	O
representations	O
for	O
students	O
.	O
Recently	O
,	O
SAINT+	O
(	O
Shin	O
et	O
al	O
.	O
,	O
2020	O
)	O
showed	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
performance	O
on	O
the	O
popular	O
EdNet	O
KT	O
task	O
using	O
a	O
Transformer	O
model	O
to	O
capture	O
temporal	O
information	O
across	O
activities	O
,	O
motivating	O
our	O
use	O
of	O
Transformer	O
LMs	O
.	O

Controllable	O
Text	O
Generation	O
aims	O
to	O
steer	O

LMs	O
towards	O
desired	O
attributes	O
.	O
Examples	O
include	O
using	O
reinforcement	O
learning	O
to	O
control	O
quality	O
metrics	O
(	O
Ranzato	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
adjusting	O
sampling	O
weights	O
to	O
control	O
for	O
poetry	O
style	O
(	O
Ghazvininejad	O
et	O
al	O
.	O
,	O
2017	O
)	O
,	O
and	O
learning	O
to	O
condition	O
on	O
valence	O
or	O
domain	O
-	O
specific	O
codes	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019;Peng	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
To	O
the	O
best	O
of	O
our	O
knowledge	O
,	O
we	O
are	O

Method	O

Given	O
any	O
autoregressive	O
language	O
model	O
(	O
e.g.	O
GPT-2	O
(	O
Radford	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
we	O
can	O
fine	O
-	O
tune	O
a	O
LM	O
-	O
KT	O
model	O
(	O
p	O
θ	O
KT	O
)	O
to	O
predict	O
whether	O
an	O
individual	O
student	O
will	O
correctly	O
answer	O
the	O
next	O
question	O
.	O
If	O
this	O
model	O
has	O
well	O
-	O
calibrated	O
uncertainty	O
,	O
we	O
can	O
use	O
its	O
predicted	O
probability	O
of	O
a	O
correct	O
answer	O
as	O
a	O
proxy	O
for	O
the	O
difficulty	O
of	O
a	O
question	O
to	O
a	O
student	O
.	O
We	O
then	O
train	O
a	O
question	O
generation	O
model	O
(	O
p	O
θ	O
QG	O
)	O
to	O
generate	O
a	O
new	O
question	O
conditioned	O
on	O
a	O
student	O
and	O
desired	O
target	O
difficulty	O
.	O

Question	O
Representation	O
Unlike	O
standard	O
DKT	O
,	O
which	O
treats	O
questions	O
as	O
IDs	O
or	O
simple	O
handcrafted	O
features	O
,	O
we	O
represent	O
questions	O
fully	O
in	O
text	O
(	O
e.g.	O
"	O
she	O
eats	O
"	O
in	O
Figure	O
1	O
)	O
.	O
This	O
is	O
a	O
key	O
contribution	O
of	O
our	O
work	O
,	O
required	O
by	O
our	O
eventual	O
goal	O
of	O
generating	O
questions	O
in	O
text	O
,	O
and	O
allows	O
the	O
model	O
to	O
leverage	O
similarity	O
across	O
linguistic	O
features	O
.	O
We	O
thus	O
represent	O
a	O
question	O
q	O
as	O
a	O
sequence	O
of	O
words	O
,	O
with	O
prefix	O
and	O
suffix	O
tokens	O
:	O

q	O
i	O
=	O
<	O
Q	O
>	O
w	O
i	O
1	O
w	O
i	O
2	O
w	O
i	O
3	O
...	O
w	O
i	O
n	O
<	O
A	O
>	O
Student	O
State	O

We	O
represent	O
a	O
student	O
as	O
a	O
temporally	O
-	O
evolving	O
sequence	O
of	O
questions	O
and	O
their	O
responses	O
.	O
As	O
in	O
much	O
previous	O
KT	O
work	O
,	O
we	O
represent	O
the	O
student	O
response	O
as	O
simply	O
correct	O
/	O
incorrect	O
,	O
with	O
special	O
tokens	O
<	O
Y	O
>	O
and	O
<	O
N	O
>	O
.	O
A	O
student	O
's	O
current	O
state	O
is	O
thus	O
represented	O
as	O
a	O
sequence	O
of	O
all	O
past	O
question	O
and	O
response	O
pairs	O
:	O

s	O
j	O
=	O
q	O
j	O
1	O
a	O
j	O
1	O
q	O
j	O
2	O
a	O
j	O
2	O
.	O

..	O
q	O
j	O
m	O
a	O
j	O
m	O
,	O
a	O
i	O
∈	O
{	O
<	O
Y>,<N	O
>	O
}	O
LM	O
-	O
KT	O
Given	O
the	O
sequential	O
nature	O
of	O
student	O
learning	O
over	O
time	O
,	O
we	O
can	O
easily	O
frame	O
knowledge	O
tracing	O
as	O
an	O
autoregressive	O
language	O
modeling	O
task	O
.	O
Given	O
a	O
dataset	O
D	O
of	O
students	O
s	O
1	O
,	O
s	O
2	O
,	O
...	O
,	O
s	O
|D|	O
,	O
we	O
employ	O
the	O
standard	O
training	O
objective	O
of	O
finding	O
the	O
parameters	O
θ	O
KT	O
that	O
minimizes	O

L	O
KT	O
=	O
−	O
|D|	O
i=1	O
|x	O
(	O
i	O
)	O
|	O
t=1	O
logp	O
θ	O
KT	O
(	O
x	O
(	O
i	O
)	O
t	O
|x	O
(	O
i	O
)	O
<	O
t	O
)	O
(	O
1	O
)	O

where	O

x	O
(	O
j	O
)	O
=	O
(	O
x	O
(	O
j	O
)	O
1	O
,	O
....	O
,	O
x	O
(	O
j	O
)	O

|x|	O
)	O
is	O
the	O
entire	O
sequence	O
tokens	O
corresponding	O
to	O
student	O
s	O
j	O
,	O
consisting	O
of	O
all	O
their	O
past	O
questions	O
and	O
answers	O
.	O
Using	O
the	O
softmax	O
output	O
of	O
the	O
LM	O
-	O
KT	O
model	O
(	O
p	O
θ	O
KT	O
)	O
,	O
we	O
estimate	O
a	O
student	O
's	O
(	O
inverse	O
)	O
difficulty	O
in	O
answering	O
a	O
specific	O
question	O
as	O
d	O
qs	O
=	O
p	O
θ	O
KT	O
(	O
<	O
Y>|s	O
,	O
q	O
)	O
.	O
We	O
find	O
that	O
p	O
θ	O
KT	O
is	O
well	O
-	O
calibrated	O
(	O
Section	O
4.2	O
)	O
,	O
yielding	O
a	O
good	O
proxy	O
for	O
the	O
true	O
question	O
difficulty	O
.	O

Question	O
Generation	O
We	O
frame	O
question	O
generation	O
as	O
finetuning	O
a	O
new	O
autoregressive	O
LM	O
.	O
Given	O
random	O
samples	O
of	O
students	O
and	O
questions	O
from	O
a	O
held	O
-	O
out	O
set	O
not	O
used	O
to	O
train	O
LM	O
-	O
KT	O
,	O
we	O
can	O
construct	O
a	O
new	O
dataset	O
D	O
consisting	O
of	O
s	O
i	O
d	O
i	O
<	O
G	O
>	O
q	O
i	O
sequences	O
,	O
where	O
<	O
G	O
>	O
is	O
a	O
special	O
generation	O
token	O
and	O
d	O
i	O
=	O
p	O
θ	O
KT	O
(	O
<	O
Y>|s	O
i	O
,	O
q	O
i	O
)	O
is	O
the	O
continuous	O
difficulty	O
value	O
assigned	O
by	O
LM	O
-	O
KT	O
.	O
We	O
learn	O
a	O
linear	O
layer	O
to	O
map	O
the	O
continuous	O
input	O
difficulty	O
into	O
a	O
difficulty	O
control	O
vector	O
c	O
d	O
of	O
dimension	O
matching	O
the	O
LM	O
word	O
-	O
embeddings	O
,	O
which	O
we	O
append	O
to	O
the	O
token	O
embeddings	O
.	O
Unlike	O
LM	O
-	O
KT	O
,	O
we	O
train	O
our	O
question	O
generation	O
model	O
p	O
θ	O
QG	O
to	O
minimize	O
the	O
loss	O
only	O
on	O
the	O
question	O
text	O
,	O
which	O
only	O
appears	O
after	O
the	O
<	O
G	O
>	O
token	O
.	O
If	O
t	O
g	O
is	O
the	O
token	O
index	O
of	O
<	O
G	O
>	O
,	O
then	O
our	O
modified	O
loss	O
is	O
:	O

L	O
QG	O
=	O
−	O
|D	O
|	O
i=1	O
|x	O
(	O
i	O
)	O
|	O
t	O
=	O
tg+1	O
logp	O
θ	O
QG	O
(	O
x	O
(	O
i	O
)	O
t	O
|x	O
(	O
i	O
)	O
<	O
t	O
)	O
(	O
2	O
)	O

where	O
sequence	O
x	O
(	O
j	O
)	O
contains	O
the	O
full	O
s	O
j	O
d	O
j	O
<	O
G	O
>	O
q	O
j	O
sequence	O
.	O
At	O
test	O
time	O
,	O
we	O
generate	O
tokens	O
w	O
1	O
...	O
w	O
n	O
conditioned	O
on	O
the	O
s	O
j	O
d	O
j	O
<	O
G	O
>	O
prefix	O
.	O

Experiments	O

Our	O
method	O
generalizes	O
to	O
any	O
education	O
activity	O
that	O
can	O
be	O
represented	O
with	O
text	O
sequences	O
.	O
Due	O
to	O
the	O
availability	O
of	O
real	O
student	O
learning	O
data	O
,	O
we	O
focus	O
on	O
a	O
reverse	O
language	O
translation	O
task	O
,	O
where	O
a	O
student	O
translates	O
phrases	O
from	O
their	O
native	O
language	O
(	O
e.g.	O
English	O
,	O
"	O
she	O
eats	O
"	O
)	O
to	O
the	O
second	O
language	O
they	O
are	O
learning	O
(	O
e.g.	O
Spanish	O
,	O
"	O
ella	O
come	O
"	O
)	O
.	O

Experimental	O
Details	O

We	O
use	O
the	O
2018	O
Duolingo	O
Shared	O
Task	O
on	O
Second	O
Language	O
Acquisition	O
Modeling	O
(	O
Settles	O
et	O
al	O
.	O
,	O
2018	O
)	O
dataset	O
,	O
which	O
contains	O
questions	O
and	O
responses	O
for	O
Duolingo	O
users	O
over	O
the	O
first	O
30	O
days	O
of	O
learning	O
a	O
second	O
language	O
.	O
While	O
the	O
original	O
task	O
's	O
goal	O
was	O
to	O
identify	O
token	O
-	O
level	O
mistakes	O
,	O
we	O
collapse	O
these	O
errors	O
into	O
binary	O
(	O
correct	O
/	O
incorrect	O
)	O
per	O
-	O
question	O
labels	O
.	O
We	O
use	O
the	O
provided	O
train	O
/	O
dev	O
/	O
test	O
splits	O
for	O
users	O
learning	O
Spanish	O
and	O
French	O
.	O
We	O
create	O
separate	O
held	O
-	O
out	O
sets	O
from	O
the	O
test	O
set	O
to	O
evaluate	O
the	O
LM	O
-	O
KT	O
and	O
question	O
generation	O
models	O
.	O
For	O
both	O
models	O
,	O
we	O
finetune	O
separate	O
GPT-2	O
(	O
Radford	O
et	O
al	O
.	O
,	O
2019	O
)	O

Results	O
:	O
Student	O
Modeling	O

We	O
evaluate	O
LM	O
-	O
KT	O
two	O
ways	O
:	O
first	O
,	O
its	O
ability	O
to	O
predict	O
if	O
an	O
individual	O
student	O
will	O
answer	O
a	O
novel	O
question	O
correctly	O
on	O
a	O
held	O
-	O
out	O
test	O
set	O
of	O
real	O
Duolingo	O
student	O
responses	O
.	O
Second	O
,	O
how	O
wellcalibrated	O
these	O
predictions	O
are	O
,	O
which	O
is	O
crucial	O
to	O
our	O
later	O
use	O
of	O
LM	O
-	O
KT	O
for	O
question	O
generation	O
.	O
Table	O
1	O
compares	O
AUC	O
-	O
ROC	O
on	O
a	O
held	O
-	O
out	O
test	O
set	O
for	O
our	O
LM	O
-	O
KT	O
model	O
with	O
standard	O
DKT	O
,	O
which	O
uses	O
question	O
IDs	O
instead	O
of	O
text	O
,	O
and	O
a	O
baseline	O
that	O
ignores	O
the	O
student	O
state	O
,	O
only	O
using	O
the	O
question	O
text	O
representation	O
.	O
This	O
question	O
only	O
baseline	O
would	O
perform	O
well	O
if	O
the	O
Duolingo	O
dataset	O
largely	O
consisted	O
of	O
universally	O
"	O
easy	O
"	O
and	O
"	O
difficult	O
"	O
questions	O
,	O
independent	O
of	O
individual	O
student	O
.	O
Our	O
results	O
show	O
that	O
incorporating	O
the	O
student	O
state	O
is	O
crucial	O
for	O
accurately	O
predicting	O
Duolingo	O
user	O
responses	O
,	O
and	O
including	O
question	O
text	O
also	O
leads	O
to	O
a	O
significant	O
improvement	O
.	O
LM	O
-	O
KT	O
outperforms	O
Standard	O
DKT	O
especially	O
on	O
novel	O
questions	O
-	O
a	O
necessary	O
generalization	O
ability	O
for	O
generation	O
.	O

Finally	O
,	O
we	O
measure	O
the	O
calibration	O
of	O
our	O
LM	O
-	O
KT	O
models	O
for	O
both	O
Spanish	O
and	O
French	O
(	O
from	O
En	O
-	O
glish	O
)	O
learners	O
,	O
which	O
is	O
the	O
crucial	O
property	O
for	O
our	O
downstream	O
generation	O
task	O
.	O
We	O
bin	O
our	O
test	O
data	O
by	O
predicted	O
question	O
difficulty	O
,	O
and	O
plot	O
the	O
fraction	O
of	O
true	O
correct	O
answers	O
in	O
each	O
bin	O
.	O
Figure	O
2	O
shows	O
that	O
LM	O
-	O
KT	O
is	O
well	O
-	O
calibrated	O
,	O
for	O
both	O
Spanish	O
and	O
French	O
,	O
meaning	O
the	O
predicted	O
difficulty	O
matches	O
the	O
empirically	O
observed	O
proportion	O
of	O
correct	O
answers	O
.	O

Results	O
:	O
Question	O
Generation	O

We	O
evaluate	O
four	O
different	O
aspects	O
of	O
our	O
question	O
generation	O
model	O
:	O
(	O
i	O
)	O
successful	O
control	O
for	O
difficulty	O
,	O
(	O
ii	O
)	O
novelty	O
,	O
(	O
iii	O
)	O
fluency	O
,	O
and	O
(	O
iv	O
)	O
latency	O
.	O

Difficulty	O
Control	O
To	O
explore	O
whether	O
our	O
question	O
generation	O
model	O
indeed	O
depends	O
on	O
target	O
difficulty	O
and	O
the	O
individual	O
student	O
,	O
we	O
first	O
measure	O
the	O
model	O
's	O
perplexity	O
on	O
a	O
held	O
-	O
out	O
test	O
set	O
of	O
Duolingo	O
questions	O
,	O
compared	O
to	O
permutation	O
baselines	O
.	O
Table	O
2	O
(	O
top	O
)	O
shows	O
that	O
perplexity	O
is	O
lower	O
for	O
true	O
student	O
/	O
target	O
difficulty	O
inputs	O
than	O
when	O
either	O
or	O
both	O
of	O
these	O
are	O
permuted	O
.	O
The	O
target	O
difficulty	O
values	O
in	O
this	O
analysis	O
were	O
defined	O
by	O
the	O
LM	O
-	O
DKT	O
model	O
.	O
We	O
can	O
remove	O
this	O
dependence	O
by	O
using	O
the	O
actual	O
student	O
responses	O
from	O
Duolingo	O
:	O
we	O
set	O
the	O
target	O
difficulty	O
to	O
1	O
if	O
the	O
student	O
was	O
correct	O
and	O
0	O
otherwise	O
.	O
Table	O
2	O
(	O
bottom	O
)	O
shows	O
our	O
model	O
prefers	O
questions	O
paired	O
with	O
these	O
"	O
true	O
correctness	O
"	O
targets	O
than	O
paired	O
with	O
random	O
ones	O
.	O

To	O
evaluate	O
how	O
well	O
our	O
generation	O
model	O
achieves	O
target	O
difficulties	O
,	O
we	O
take	O
15	O
unseen	O
students	O
and	O
generate	O
30	O
questions	O
for	O
each	O
of	O
9	O
input	O
difficulties	O
(	O
0.1	O
-	O
0.9	O
)	O
.	O
We	O
then	O
use	O
LM	O
-	O
KT	O
(	O
a	O
wellcalibrated	O
proxy	O
for	O
true	O
difficulty	O
)	O
to	O
measure	O
the	O
difficulty	O
of	O
these	O
generated	O
questions	O
for	O
each	O
student	O
.	O
Figure	O
3	O
shows	O
that	O
we	O
are	O
able	O
to	O
achieve	O
fine	O
-	O
grained	O
control	O
over	O
target	O
difficulty	O
for	O
both	O
Spanish	O
and	O
French	O
students	O
,	O
with	O
an	O
average	O
Root	O
-	O
Mean	O
Squared	O
Error	O
(	O
RMSE	O
)	O
of	O
.052	O
across	O
all	O
students	O
and	O
target	O
difficulties	O
.	O
Adding	O
a	O
sampling	O
penalty	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
increases	O
the	O
variance	O
in	O
difficulty	O
(	O
RMSE	O
.062	O
)	O
in	O
exchange	O
for	O
more	O
novel	O
and	O
diverse	O
questions	O
,	O
as	O
discussed	O
next	O
.	O

Novelty	O
and	O
Fluency	O
By	O
leveraging	O
a	O
pretrained	O
language	O
model	O
's	O
ability	O
to	O
manipulate	O
structure	O
,	O
we	O
can	O
generate	O
novel	O
questions	O
not	O
present	O
in	O
the	O
entire	O
Duolingo	O
question	O
set	O
(	O
See	O
Table	O
3	O
)	O
.	O
Across	O
4,050	O
questions	O
generated	O
for	O
Spanish	O
learners	O
,	O
we	O
found	O
that	O
with	O
a	O
repetition	O
penalty	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
around	O
43	O
%	O
of	O
all	O
questions	O
,	O
and	O
66	O
%	O
of	O
high	O
difficulty	O
(	O
d	O
=	O
0.1	O
)	O
required	O
to	O
rank	O
all	O
questions	O
in	O
the	O
pool	O
,	O
varying	O
its	O
size	O
(	O
Figure	O
4	O
)	O
.	O
On	O
one	O
NVIDIA	O
Titan	O
XP	O
GPU	O
,	O
we	O
find	O
that	O
,	O
averaged	O
across	O
all	O
target	O
difficulties	O
,	O
our	O
question	O
generation	O
model	O
takes	O
half	O
the	O
time	O
to	O
achieve	O
the	O
same	O
quality	O
as	O
pool	O
selection	O
.	O
The	O
gap	O
increases	O
when	O
trying	O
to	O
sample	O
harder	O
questions	O
(	O
d	O
<	O
0.5	O
)	O
-even	O
a	O
pool	O
size	O
of	O
1000	O
does	O
not	O
have	O
sufficient	O
difficult	O
questions	O
,	O
likely	O
due	O
to	O
a	O
skew	O
in	O
the	O
Duolingo	O
question	O
set	O
.	O
Additional	O
controls	O
,	O
such	O
as	O
for	O
style	O
or	O
topic	O
,	O
can	O
easily	O
be	O
combined	O
with	O
our	O
generation	O
method	O
,	O
but	O
would	O
make	O
pool	O
selection	O
exponentially	O
more	O
complex	O
.	O
Pool	O
Sampling	O
(	O
all	O
targets	O
)	O
Pool	O
Sampling	O
(	O
difficult	O
targets	O
only	O
)	O
Generation	O
(	O
all	O
targets	O
)	O
Generation	O
(	O
difficult	O
targets	O
only	O
)	O

Figure	O
4	O
:	O
Pool	O
selection	O
(	O
for	O
one	O
student	O
)	O
suffers	O
worse	O
question	O
quality	O
vs.	O
latency	O
trade	O
-	O
off	O
than	O
question	O
generation	O
,	O
especially	O
for	O
sampling	O
difficult	O
questions	O
.	O

Conclusion	O

Our	O
work	O
is	O
a	O
first	O
step	O
toward	O
showing	O
that	O
sequence	O
-	O
based	O
models	O
combined	O
with	O
domain	O
knowledge	O
,	O
such	O
as	O
pre	O
-	O
trained	O
LMs	O
,	O
can	O
be	O
leveraged	O
for	O
adaptive	O
learning	O
tasks	O
.	O
We	O
show	O
how	O
to	O
use	O
modern	O
LMs	O
to	O
generate	O
novel	O
reversetranslation	O
questions	O
that	O
achieve	O
a	O
target	O
difficulty	O
,	O
allowing	O
adaptive	O
education	O
methods	O
to	O
expand	O
beyond	O
limited	O
question	O
pools	O
.	O
Limitations	O
of	O
our	O
approach	O
include	O
the	O
compute	O
constraints	O
of	O
large	O
LMs	O
and	O
training	O
data	O
availability	O
.	O
More	O
detailed	O
student	O
data	O
will	O
be	O
crucial	O
to	O
future	O
model	O
development	O
.	O
For	O
instance	O
,	O
while	O
most	O
publicly	O
available	O
education	O
datasets	O
do	O
not	O
include	O
the	O
full	O
student	O
responses	O
(	O
e.g.	O
full	O
translation	O
response	O
in	O
Duolingo	O
)	O
,	O
such	O
information	O
could	O
significantly	O
improve	O
the	O
performance	O
of	O
our	O
LM	O
-	O
KT	O
model	O
.	O
Other	O
future	O
directions	O
include	O
exploring	O
non	O
-	O
language	O
domains	O
,	O
such	O
as	O
math	O
or	O
logic	O
exercises	O
,	O
and	O
controlling	O
for	O
auxiliary	O
objectives	O
such	O
as	O
question	O
topic	O
.	O

Finally	O
,	O
designing	O
appropriate	O
user	O
studies	O
to	O
evaluate	O
our	O
method	O
is	O
a	O
complex	O
yet	O
critical	O
next	O
step	O
to	O
determine	O
its	O
suitability	O
in	O
a	O
real	O
-	O
world	O
education	O
setting	O
.	O
Our	O
techniques	O
allows	O
control	O
for	O
individual	O
student	O
difficulty	O
,	O
but	O
it	O
leaves	O
open	O
the	O
question	O
of	O
optimal	O
curriculum	O
design	O
using	O
difficulty	O
-	O
directed	O
question	O
generation	O
.	O

Broader	O
Impact	O

Online	O
education	O
platforms	O
can	O
increase	O
the	O
accessibility	O
of	O
high	O
quality	O
educational	O
resources	O
for	O
students	O
around	O
the	O
world	O
.	O
Adaptive	O
techniques	O
that	O
allow	O
for	O
more	O
individualized	O
learning	O
strategies	O
can	O
help	O
such	O
technologies	O
be	O
more	O
inclusive	O
for	O
students	O
who	O
make	O
less	O
-	O
common	O
mistakes	O
or	O
have	O
different	O
prior	O
backgrounds	O
(	O
Lee	O
and	O
Brunskill	O
,	O
2012	O
)	O
.	O
However	O
,	O
our	O
method	O
is	O
subject	O
to	O
biases	O
found	O
in	O
the	O
training	O
data	O
,	O
and	O
careful	O
consideration	O
of	O
using	O
safe	O
and	O
appropriate	O
data	O
is	O
crucial	O
in	O
an	O
education	O
context	O
.	O
Moreover	O
,	O
our	O
specific	O
use	O
of	O
pre	O
-	O
trained	O
LMs	O
relies	O
on	O
the	O
significant	O
progress	O
of	O
NLP	O
tools	O
for	O
English	O
language	O
-further	O
research	O
and	O
development	O
of	O
these	O
tools	O
for	O
other	O
languages	O
can	O
help	O
ensure	O
our	O
method	O
benefits	O
a	O
larger	O
population	O
of	O
students	O
.	O

A	O
APPENDIX	O

A.1	O
Dataset	O
Details	O

The	O
2018	O
Duolingo	O
Shared	O
Task	O
on	O
Second	O
Language	O
Acquisition	O
Modeling	O
(	O
Settles	O
et	O
al	O
.	O
,	O
2018	O
)	O
dataset	O
contains	O
questions	O
and	O
responses	O
for	O
Duolingo	O
users	O
over	O
the	O
first	O
30	O
days	O
of	O
learning	O
a	O
second	O
language	O
.	O
The	O
dataset	O
contains	O
three	O
different	O
question	O
types	O
:	O
reverse	O
translate	O
(	O
free	O
response	O
translation	O
of	O
a	O
given	O
prompt	O
in	O
the	O
language	O
they	O
are	O
learning	O
)	O
,	O
reverse	O
tap	O
(	O
a	O
selection	O
-	O
based	O
equivalent	O
of	O
reverse	O
translate	O
)	O
,	O
and	O
listen	O
,	O
where	O
students	O
listen	O
to	O
a	O
vocal	O
utterance	O
.	O
We	O
focus	O
on	O
the	O
reverse	O
translate	O
question	O
type	O
for	O
English	O
-	O
speaking	O
students	O
learning	O
French	O
and	O
Spanish	O
.	O
The	O
dataset	O
size	O
for	O
French	O
learners	O
(	O
1.2k	O
users	O
)	O
is	O
roughly	O
half	O
the	O
size	O
of	O
that	O
for	O
Spanish	O
learners	O
(	O
2.6k	O
users	O
)	O
.	O

Because	O
the	O
original	O
dataset	O
was	O
intended	O
for	O
per	O
-	O
token	O
error	O
prediction	O
,	O
each	O
question	O
has	O
per	O
-	O
token	O
information	O
that	O
includes	O
whether	O
the	O
student	O
translated	O
the	O
token	O
correctly	O
,	O
as	O
well	O
as	O
Universal	O
Dependencies	O
tags	O
such	O
as	O
part	O
of	O
speech	O
and	O
morphology	O
labels	O
.	O
We	O
use	O
the	O
full	O
question	O
text	O
,	O
rather	O
than	O
individual	O
tokens	O
,	O
for	O
our	O
task	O
,	O
and	O
combine	O
the	O
labels	O
such	O
that	O
if	O
a	O
Duolingo	O
user	O
incorrectly	O
translated	O
one	O
or	O
more	O
tokens	O
in	O
a	O
question	O
,	O
the	O
entire	O
question	O
is	O
marked	O
incorrect	O
.	O
We	O
do	O
not	O
use	O
any	O
additional	O
features	O
.	O

We	O
use	O
the	O
publicly	O
provided	O
train	O
/	O
dev	O
/	O
test	O
splits	O
from	O
the	O
Shared	O
Task	O
,	O
which	O
are	O
temporally	O
ordered	O
in	O
sequence	O
.	O
We	O
therefore	O
construct	O
student	O
states	O
by	O
tracking	O
user	O
IDs	O
throughout	O
the	O
datasets	O
and	O
appending	O
each	O
new	O
question	O
and	O
response	O
to	O
the	O
current	O
student	O
state	O
.	O
When	O
evaluating	O
our	O
LM	O
-	O
KT	O
model	O
,	O
we	O
use	O
the	O
true	O
responses	O
of	O
preceding	O
questions	O
in	O
the	O
test	O
set	O
to	O
form	O
the	O
student	O
state	O
for	O
a	O
given	O
question	O
.	O
Overall	O
,	O
we	O
find	O
that	O
the	O
dataset	O
is	O
severely	O
imbalanced	O
(	O
as	O
in	O
the	O
original	O
task	O
)	O
-about	O
30	O
%	O
of	O
questions	O
are	O
answered	O
incorrectly	O
across	O
students	O
studying	O
both	O
French	O
and	O
Spanish	O
.	O

Finally	O
,	O
we	O
create	O
a	O
held	O
-	O
out	O
set	O
of	O
Duolingo	O
questions	O
for	O
both	O
French	O
and	O
Spanish	O
learners	O
to	O
create	O
the	O
training	O
data	O
for	O
our	O
question	O
generation	O
model	O
.	O
From	O
a	O
set	O
of	O
random	O
student	O
states	O
,	O
we	O
select	O
questions	O
from	O
this	O
set	O
and	O
use	O
a	O
trained	O
LM	O
-	O
KT	O
model	O
to	O
assign	O
the	O
difficulty	O
score	O
.	O
In	O
practice	O
,	O
this	O
held	O
-	O
out	O
set	O
can	O
come	O
from	O
any	O
source	O
,	O
not	O
just	O
Duolingo	O
data	O
.	O

A.2	O
Model	O
Training	O
Details	O

To	O
train	O
both	O
our	O
LM	O
-	O
KT	O
knowledge	O
tracing	O
model	O
and	O
our	O
question	O
generation	O
model	O
,	O
we	O
use	O
the	O
pre	O
-	O
trained	O
OpenAI	O
GPT-2	O
model	O
from	O
the	O
HuggingFace	O
Transformers	O
library	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
For	O
question	O
generation	O
,	O
we	O
modify	O
the	O
library	O
to	O
add	O
a	O
linear	O
layer	O
and	O
the	O
modified	O
loss	O
function	O
for	O
question	O
generation	O
from	O
Section	O
3	O
.	O

We	O
use	O
1	O
NVIDIA	O
TitanXP	O
GPU	O
with	O
12	O
GB	O
of	O
memory	O
available	O
.	O
Because	O
the	O
maximum	O
input	O
sequence	O
length	O
of	O
the	O
GPT-2	O
model	O
we	O
use	O
is	O
1024	O
tokens	O
,	O
we	O
resize	O
all	O
inputs	O
to	O
the	O
last	O
1024	O
tokens	O
before	O
training	O
.	O
We	O
report	O
results	O
for	O
an	O
LM	O
-	O
KT	O
model	O
trained	O
for	O
13k	O
steps	O
with	O
the	O
default	O
batch	O
size	O
of	O
2	O
and	O
learning	O
rate	O
of	O
5e-5	O
,	O
and	O
a	O
Question	O
Generation	O
model	O
trained	O
for	O
25k	O
steps	O
with	O
the	O
same	O
batch	O
size	O
and	O
learning	O
rate	O
.	O
The	O
total	O
compute	O
time	O
to	O
train	O
both	O
models	O
was	O
2.5	O
hours	O
for	O
each	O
language	O
learning	O
task	O
.	O

A.3	O
Question	O
Generation	O
Details	O

For	O
both	O
French	O
and	O
Spanish	O
question	O
generation	O
models	O
,	O
we	O
select	O
15	O
students	O
unseen	O
during	O
training	O
and	O
generate	O
30	O
questions	O
across	O
9	O
difficulties	O
from	O
0.1	O
to	O
0.9	O
,	O
using	O
nucleus	O
sampling	O
(	O
Holtzman	O
et	O
al	O
.	O
,	O
2020	O
)	O
(	O
p	O
=	O
0.99	O
)	O
with	O
a	O
maximum	O
output	O
length	O
of	O
20	O
tokens	O
.	O
We	O
also	O
vary	O
a	O
repetition	O
penalty	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
that	O
penalizes	O
for	O
previous	O
tokens	O
(	O
including	O
those	O
in	O
the	O
student	O
state	O
)	O
.	O
Lastly	O
,	O
we	O
resize	O
all	O
prompts	O
(	O
student	O
state	O
and	O
target	O
difficulty	O
)	O
to	O
fit	O
into	O
the	O
GPT-2	O
Model	O
by	O
taking	O
the	O
most	O
recent	O
1024	O
tokens	O
,	O
as	O
in	O
training	O
.	O
This	O
is	O
a	O
limitation	O
of	O
our	O
work	O
,	O
as	O
the	O
full	O
student	O
history	O
is	O
not	O
able	O
to	O
be	O
considered	O
for	O
students	O
who	O
have	O
answered	O
a	O
large	O
set	O
of	O
questions	O
.	O

A.4	O
Additional	O
Question	O
Generation	O
Outputs	O

Our	O
question	O
generation	O
model	O
demonstrates	O
the	O
ability	O
to	O
generate	O
novel	O
questions	O
that	O
do	O
not	O
exist	O
in	O
the	O
entire	O
Duolingo	O
question	O
dataset	O
,	O
especially	O
when	O
a	O
sampling	O
penalty	O
is	O
applied	O
to	O
encourage	O
more	O
diverse	O
outputs	O
.	O
However	O
,	O
this	O
comes	O
at	O
a	O
cost	O
to	O
fluency	O
.	O
Below	O
we	O
include	O
a	O
set	O
of	O
outputs	O
generated	O
by	O
our	O
model	O
for	O
1	O
Spanish	O
student	O
and	O
1	O
French	O
student	O
from	O
the	O
Duolingo	O
dataset	O
,	O
with	O
a	O
target	O
difficulty	O
of	O
d	O
=	O
0.1	O
,	O
and	O
both	O
with	O
and	O
without	O
a	O
repetition	O
penalty	O
.	O
We	O
observe	O
that	O
while	O
applying	O
a	O
penalty	O
results	O
in	O
a	O
far	O
more	O
novel	O
questions	O
generated	O
,	O
several	O
of	O
these	O
are	O
also	O
non	O
-	O
fluent	O
,	O
using	O
a	O
combination	O
of	O
manual	O
judgement	O
and	O
the	O
Python	O
language	O
-	O
check	O
package	O
(	O
https://pypi.org/project/language-check/	O
)	O
.	O

An	O
Exploratory	O
Analysis	O
of	O
Multilingual	O
Word	O
-	O
Level	O
Quality	O
Estimation	O
with	O
Cross	O
-	O
Lingual	O
Transformers	O

Most	O
studies	O
on	O
word	O
-	O
level	O
Quality	O
Estimation	O
(	O
QE	O
)	O
of	O
machine	O
translation	O
focus	O
on	O
languagespecific	O
models	O
.	O
The	O
obvious	O
disadvantages	O
of	O
these	O
approaches	O
are	O
the	O
need	O
for	O
labelled	O
data	O
for	O
each	O
language	O
pair	O
and	O
the	O
high	O
cost	O
required	O
to	O
maintain	O
several	O
language	O
-	O
specific	O
models	O
.	O
To	O
overcome	O
these	O
problems	O
,	O
we	O
explore	O
different	O
approaches	O
to	O
multilingual	O
,	O
word	O
-	O
level	O
QE	O
.	O
We	O
show	O
that	O
multilingual	O
QE	O
models	O
perform	O
on	O
par	O
with	O
the	O
current	O
language	O
-	O
specific	O
models	O
.	O
In	O
the	O
cases	O
of	O
zeroshot	O
and	O
few	O
-	O
shot	O
QE	O
,	O
we	O
demonstrate	O
that	O
it	O
is	O
possible	O
to	O
accurately	O
predict	O
word	O
-	O
level	O
quality	O
for	O
any	O
given	O
new	O
language	O
pair	O
from	O
models	O
trained	O
on	O
other	O
language	O
pairs	O
.	O
Our	O
findings	O
suggest	O
that	O
the	O
word	O
-	O
level	O
QE	O
models	O
based	O
on	O
powerful	O
pre	O
-	O
trained	O
transformers	O
that	O
we	O
propose	O
in	O
this	O
paper	O
generalise	O
well	O
across	O
languages	O
,	O
making	O
them	O
more	O
useful	O
in	O
real	O
-	O
world	O
scenarios	O
.	O

Quality	O
Estimation	O
(	O
QE	O
)	O
is	O
the	O
task	O
of	O
assessing	O
the	O
quality	O
of	O
a	O
translation	O
without	O
having	O
access	O
to	O
a	O
reference	O
translation	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2009	O
)	O
.	O
Translation	O
quality	O
can	O
be	O
estimated	O
at	O
different	O
levels	O
of	O
granularity	O
:	O
word	O
,	O
sentence	O
and	O
document	O
level	O
(	O
I	O
ve	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
So	O
far	O
the	O
most	O
popular	O
task	O
has	O
been	O
sentence	O
-	O
level	O
QE	O
,	O
in	O
which	O
QE	O
models	O
provide	O
a	O
score	O
for	O
each	O
pair	O
of	O
source	O
and	O
target	O
sentences	O
.	O
A	O
more	O
challenging	O
task	O
,	O
which	O
is	O
currently	O
receiving	O
a	O
lot	O
of	O
attention	O
from	O
the	O
research	O
community	O
,	O
is	O
word	O
-	O
level	O
quality	O
estimation	O
.	O
This	O
task	O
provides	O
more	O
fine	O
-	O
grained	O
information	O
about	O
the	O
quality	O
of	O
a	O
translation	O
,	O
indicating	O
which	O
words	O
from	O
the	O
source	O
have	O
been	O
incorrectly	O
translated	O
in	O
the	O
target	O
,	O
and	O
whether	O
the	O
words	O
inserted	O
between	O
these	O
words	O
are	O
correct	O
(	O
good	O
vs	O
bad	O
gaps	O
)	O
.	O
This	O
information	O
can	O
be	O
useful	O
for	O
post	O
-	O
editors	O
by	O
indicating	O
the	O
parts	O
of	O
a	O
sentence	O
on	O
which	O
they	O
have	O
to	O
focus	O
more	O
.	O

Word	O
-	O
level	O
QE	O
is	O
generally	O
framed	O
as	O
a	O
supervised	O
ML	O
problem	O
(	O
Kepler	O
et	O
al	O
.	O
,	O
2019;Lee	O
,	O
2020	O
)	O
trained	O
on	O
data	O
in	O
which	O
the	O
correctness	O
of	O
translation	O
is	O
labelled	O
at	O
word	O
-	O
level	O
(	O
i.e.	O
good	O
,	O
bad	O
,	O
gap	O
)	O
.	O

The	O
training	O
data	O
publicly	O
available	O
to	O
build	O
wordlevel	O
QE	O
models	O
is	O
limited	O
to	O
very	O
few	O
language	O
pairs	O
,	O
which	O
makes	O
it	O
difficult	O
to	O
build	O
QE	O
models	O
for	O
many	O
languages	O
.	O
From	O
an	O
application	O
perspective	O
,	O
even	O
for	O
the	O
languages	O
with	O
resources	O
,	O
it	O
is	O
difficult	O
to	O
maintain	O
separate	O
QE	O
models	O
for	O
each	O
language	O
since	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
QE	O
models	O
are	O
large	O
in	O
size	O
(	O
Ranasinghe	O
et	O
al	O
.	O
,	O
2020b	O
)	O
.	O

In	O
our	O
paper	O
,	O
we	O
address	O
this	O
problem	O
by	O
developing	O
multilingual	O
word	O
-	O
level	O
QE	O
models	O
which	O
perform	O
competitively	O
in	O
different	O
domains	O
,	O
MT	O
types	O
and	O
language	O
pairs	O
.	O
In	O
addition	O
,	O
for	O
the	O
first	O
time	O
,	O
we	O
propose	O
word	O
-	O
level	O
QE	O
as	O
a	O
zero	O
-	O
shot	O
crosslingual	O
transfer	O
task	O
,	O
enabling	O
new	O
avenues	O
of	O
research	O
in	O
which	O
multilingual	O
models	O
can	O
be	O
trained	O
once	O
and	O
then	O
serve	O
a	O
multitude	O
of	O
languages	O
and	O
domains	O
.	O
The	O
main	O
contributions	O
of	O
this	O
paper	O
are	O
the	O
following	O
:	O
i	O
We	O
introduce	O
a	O
simple	O
architecture	O
to	O
perform	O
word	O
-	O
level	O
quality	O
estimation	O
that	O
predicts	O
the	O
quality	O
of	O
the	O
words	O
in	O
the	O
source	O
sentence	O
,	O
target	O
sentence	O
and	O
the	O
gaps	O
in	O
the	O
target	O
sentence	O
.	O

ii	O
We	O
explore	O
multilingual	O
,	O
word	O
-	O
level	O
quality	O
estimation	O
with	O
the	O
proposed	O
architecture	O
.	O
We	O
show	O
that	O
multilingual	O
models	O
are	O
competitive	O
with	O
bilingual	O
models	O
.	O

iii	O
We	O
inspect	O
few	O
-	O
shot	O
and	O
zero	O
-	O
shot	O
word	O
-	O
level	O
quality	O
estimation	O
with	O
the	O
bilingual	O
and	O
multilingual	O
models	O
.	O
We	O
report	O
how	O
the	O
sourcetarget	O
direction	O
,	O
domain	O
and	O
MT	O
type	O
affect	O
the	O
predictions	O
for	O
a	O
new	O
language	O
pair	O
.	O

iv	O
We	O
release	O
the	O
code	O
and	O
the	O
pre	O
-	O
trained	O
models	O
as	O
part	O
of	O
an	O
open	O
-	O
source	O
framework	O
1	O
.	O
(	O
Kepler	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
However	O
,	O
the	O
current	O
state	O
of	O
the	O
art	O
in	O
word	O
-	O
level	O
QE	O
is	O
based	O
on	O
transformers	O
like	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
XLM	O
-	O
R	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2020	O
)	O
where	O
a	O
simple	O
linear	O
layer	O
is	O
added	O
on	O
top	O
of	O
the	O
transformer	O
model	O
to	O
obtain	O
the	O
predictions	O
(	O
Lee	O
,	O
2020	O
)	O
.	O
All	O
of	O
these	O
approaches	O
consider	O
quality	O
estimation	O
as	O
a	O
language	O
-	O
specific	O
task	O
and	O
build	O
a	O
different	O
model	O
for	O
each	O
language	O
pair	O
.	O
This	O
approach	O
has	O
many	O
drawbacks	O
in	O
real	O
-	O
world	O
applications	O
,	O
some	O
of	O
which	O
are	O
discussed	O
in	O
Section	O
1	O
.	O

Multilinguality	O
Multilinguality	O
allows	O
training	O
a	O
single	O
model	O
to	O
perform	O
a	O
task	O
from	O
and/or	O
to	O
multiple	O
languages	O
.	O
Even	O
though	O
this	O
has	O
been	O
applied	O
to	O
many	O
tasks	O
Zampieri	O
,	O
2020	O
,	O
2021	O
)	O
including	O
NMT	O
(	O
Nguyen	O
and	O
Chiang	O
,	O
2017;Aharoni	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
multilingual	O
approaches	O
have	O
been	O
rarely	O
used	O
in	O
QE	O
.	O
Shah	O
and	O
Specia	O
(	O
2016	O
)	O
explore	O
QE	O
models	O
for	O
more	O
than	O
one	O
language	O
where	O
they	O
use	O
multitask	O
learning	O
with	O
annotators	O
or	O
languages	O
as	O
multiple	O
tasks	O
.	O
They	O
show	O
that	O
multilingual	O
models	O
led	O
to	O
marginal	O
improvements	O
over	O
bilingual	O
ones	O
with	O
a	O
traditional	O
black	O
-	O
box	O
,	O
feature	O
-	O
based	O
approach	O
.	O
In	O
a	O
recent	O
study	O
,	O
Ranasinghe	O
et	O
al	O
.	O
(	O
2020b	O
)	O
show	O
that	O
multilingual	O
QE	O
models	O
based	O
on	O
transformers	O
trained	O
on	O
high	O
-	O
resource	O
languages	O
can	O
be	O
used	O
for	O
zeroshot	O
,	O
sentence	O
-	O
level	O
QE	O
in	O
low	O
-	O
resource	O
languages	O
.	O

In	O
a	O
similar	O
architecture	O
,	O
but	O
with	O
multi	O
-	O
task	O
learning	O
,	O
report	O
that	O
multilingual	O
QE	O
models	O
outperform	O
bilingual	O
models	O
,	O
particularly	O
in	O
less	O
balanced	O
quality	O
label	O
distributions	O
and	O
lowresource	O
settings	O
.	O
However	O
,	O
these	O
two	O
papers	O
are	O
focused	O
on	O
sentence	O
-	O
level	O
QE	O
and	O
to	O
the	O
best	O
of	O
our	O
knowledge	O
,	O
no	O
prior	O
work	O
has	O
been	O
done	O
on	O
multilingual	O
,	O
word	O
-	O
level	O
QE	O
models	O
.	O

In	O
our	O
experiments	O
,	O
we	O
observed	O
that	O
multilingual	O
QE	O
models	O
deliver	O
excellent	O
results	O
on	O
the	O
language	O
pairs	O
they	O
were	O
trained	O
on	O
.	O
In	O
addition	O
,	O
the	O
multilingual	O
QE	O
models	O
perform	O
well	O
in	O
the	O
majority	O
of	O
the	O
zero	O
-	O
shot	O
scenarios	O
where	O
the	O
multilingual	O
QE	O
model	O
is	O
tested	O
on	O
an	O
unseen	O
language	O
pair	O
.	O
Furthermore	O
,	O
multilingual	O
models	O
perform	O
very	O
well	O
with	O
few	O
-	O
shot	O
learning	O
on	O
an	O
unseen	O
language	O
pair	O
when	O
compared	O
to	O
training	O
from	O
scratch	O
for	O
that	O
language	O
pair	O
,	O
proving	O
that	O
multilingual	O
QE	O
models	O
are	O
effective	O
even	O
with	O
a	O
limited	O
number	O
of	O
training	O
instances	O
.	O
While	O
we	O
centered	O
our	O
analysis	O
around	O
the	O
F1	O
-	O
score	O
of	O
the	O
target	O
words	O
,	O
these	O
findings	O
are	O
consistent	O
with	O
the	O
F1	O
-	O
score	O
of	O
the	O
target	O
gaps	O
and	O
the	O
F1	O
-	O
score	O
of	O
the	O
source	O
words	O
too	O
.	O
This	O
suggests	O
that	O
we	O
can	O
train	O
a	O
single	O
multilingual	O
QE	O
model	O
on	O
as	O
many	O
languages	O
as	O
possible	O
and	O
apply	O
it	O
on	O
other	O
language	O
pairs	O
as	O
well	O
.	O
These	O
findings	O
can	O
be	O
beneficial	O
to	O
perform	O
QE	O
in	O
low	O
-	O
resource	O
languages	O
for	O
which	O
the	O
training	O
data	O
is	O
scarce	O
and	O
when	O
maintaining	O
several	O
QE	O
models	O
for	O
different	O
language	O
pairs	O
is	O
arduous	O
.	O

In	O
this	O
paper	O
,	O
we	O
explored	O
multilingual	O
,	O
word	O
-	O
level	O
QE	O
with	O
transformers	O
.	O
We	O
introduced	O
a	O
new	O
architecture	O
based	O
on	O
transformers	O
to	O
perform	O
wordlevel	O
QE	O
.	O
The	O
implementation	O
of	O
the	O
architecture	O
,	O
which	O
is	O
based	O
on	O
Hugging	O
Face	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
has	O
been	O
integrated	O
into	O
the	O
TransQuest	O
framework	O
(	O
Ranasinghe	O
et	O
al	O
.	O
,	O
2020b	O
)	O
which	O
won	O
the	O
WMT	O
2020	O
QE	O
task	O
)	O
on	O
sentencelevel	O
direct	O
assessment	O
(	O
Ranasinghe	O
et	O
al	O
.	O
,	O
2020a	O
)	O
2	O
.	O

We	O
also	O
evaluated	O
how	O
the	O
QE	O
models	O
behave	O
with	O
a	O
limited	O
number	O
of	O
training	O
instances	O
.	O
For	O
each	O
language	O
pair	O
,	O
we	O
initiated	O
the	O
weights	O
of	O
the	O
bilingual	O
model	O
with	O
those	O
of	O
the	O
relevant	O
All-1	O
QE	O
and	O
trained	O
it	O
on	O
100	O
,	O
200	O
,	O
300	O
and	O
up	O
to	O
1000	O
training	O
instances	O
.	O
We	O
compared	O
the	O
results	O
with	O
those	O
obtained	O
having	O
trained	O
the	O
QE	O
model	O
from	O
scratch	O
for	O
that	O
language	O
pair	O
.	O
The	O
results	O
in	O
Figure	O
2	O
show	O

Few	O
-	O
shot	O
QE	O

One	O
limitation	O
of	O
the	O
zero	O
-	O
shot	O
QE	O
is	O
its	O
inability	O
to	O
perform	O
when	O
the	O
language	O
direction	O
changes	O
.	O
In	O
the	O
scenario	O
where	O
we	O
performed	O
zero	O
-	O
shot	O
learning	O
from	O
De	O
-	O
En	O
to	O
other	O
language	O
pairs	O
,	O
results	O
degraded	O
considerably	O
from	O
the	O
bilingual	O
result	O
.	O
Similarly	O
,	O
the	O
performance	O
is	O
rather	O
poor	O
when	O
we	O
test	O
on	O
De	O
-	O
En	O
for	O
the	O
multilingual	O
zero	O
-	O
shot	O
experiment	O
as	O
the	O
direction	O
of	O
all	O
the	O
other	O
pairs	O
used	O
for	O
training	O
is	O
different	O
.	O
This	O
is	O
in	O
line	O
with	O
results	O
reported	O
by	O
Ranasinghe	O
et	O
al	O
.	O
(	O
2020b	O
)	O
for	O
sentence	O
level	O
.	O

We	O
also	O
experimented	O
with	O
zero	O
-	O
shot	O
QE	O
with	O
multilingual	O
QE	O
models	O
.	O
We	O
trained	O
the	O
QE	O
model	O
in	O
all	O
the	O
pairs	O
except	O
one	O
and	O
performed	O
predic	O
-	O
tion	O
on	O
the	O
test	O
set	O
of	O
the	O
language	O
pair	O
left	O
out	O
.	O
In	O
section	O
II	O
(	O
"	O
All-1	O
"	O
)	O
,	O
we	O
show	O
its	O
difference	O
to	O
the	O
multilingual	O
QE	O
model	O
.	O
This	O
also	O
provides	O
competitive	O
results	O
for	O
the	O
majority	O
of	O
the	O
languages	O
,	O
proving	O
it	O
is	O
possible	O
to	O
train	O
a	O
single	O
multilingual	O
QE	O
model	O
and	O
extend	O
it	O
to	O
a	O
multitude	O
of	O
languages	O
and	O
domains	O
.	O
This	O
approach	O
provides	O
better	O
results	O
than	O
performing	O
transfer	O
learning	O
from	O
a	O
bilingual	O
model	O
.	O

To	O
test	O
whether	O
a	O
QE	O
model	O
trained	O
on	O
a	O
particular	O
language	O
pair	O
can	O
be	O
generalised	O
to	O
other	O
language	O
pairs	O
,	O
different	O
domains	O
and	O
MT	O
types	O
,	O
we	O
performed	O
zero	O
-	O
shot	O
quality	O
estimation	O
.	O
We	O
used	O
the	O
QE	O
model	O
trained	O
on	O
a	O
particular	O
language	O
pair	O
and	O
evaluated	O
it	O
on	O
the	O
test	O
sets	O
of	O
the	O
other	O
language	O
pairs	O
.	O
Non	O
-	O
diagonal	O
values	O
of	O
section	O
I	O
in	O
Table	O
2	O
show	O
how	O
each	O
QE	O
model	O
performed	O
on	O
other	O
language	O
pairs	O
.	O
For	O
better	O
visualisation	O
,	O
the	O
nondiagonal	O
values	O
of	O
section	O
I	O
of	O
Table	O
2	O
show	O
by	O
how	O
much	O
the	O
score	O
changes	O
when	O
the	O
zero	O
-	O
shot	O
QE	O
model	O
is	O
used	O
instead	O
of	O
the	O
bilingual	O
QE	O
model	O
.	O
As	O
can	O
be	O
seen	O
,	O
the	O
scores	O
decrease	O
,	O
but	O
this	O
decrease	O
is	O
negligible	O
and	O
is	O
to	O
be	O
expected	O
.	O
For	O
most	O
pairs	O
,	O
the	O
QE	O
model	O
that	O
did	O
not	O
see	O
any	O
training	O
instances	O
of	O
that	O
particular	O
language	O
pair	O
outperforms	O
the	O
baselines	O
that	O
were	O
trained	O
extensively	O
on	O
that	O
particular	O
language	O
pair	O
.	O
Further	O
analysing	O
the	O
results	O
,	O
we	O
can	O
see	O
that	O
zero	O
-	O
shot	O
QE	O
performs	O
better	O
when	O
the	O
language	O
pair	O
shares	O
some	O
properties	O
such	O
as	O
domain	O
,	O
MT	O
type	O
or	O
language	O
direction	O
.	O
For	O
example	O
,	O
En	O
-	O
De	O
SMT	O
⇒	O
En	O
-	O
Cs	O
SMT	O
is	O
better	O
than	O
En	O
-	O
De	O
NMT	O
⇒	O
En	O
-	O
Cs	O
SMT	O
and	O
En	O
-	O
De	O
SMT	O
⇒	O
En	O
-	O
De	O
NMT	O
is	O
better	O
than	O
En	O
-	O
Cs	O
SMT	O
⇒	O
En	O
-	O
De	O
NMT	O
.	O

Zero	O
-	O
shot	O
QE	O

We	O
combined	O
instances	O
from	O
all	O
the	O
language	O
pairs	O
and	O
built	O
a	O
single	O
word	O
-	O
level	O
QE	O
model	O
.	O
Our	O
results	O
,	O
displayed	O
in	O
section	O
II	O
(	O
"	O
All	O
"	O
)	O
of	O
Table	O
2	O
,	O
show	O
that	O
multilingual	O
models	O
perform	O
on	O
par	O
with	O
bilingual	O
models	O
or	O
even	O
better	O
for	O
some	O
language	O
pairs	O
.	O
We	O
also	O
investigate	O
whether	O
combining	O
language	O
pairs	O
that	O
share	O
either	O
the	O
same	O
domain	O
or	O
MT	O
type	O
can	O
be	O
more	O
beneficial	O
,	O
since	O
it	O
is	O
possible	O
that	O
the	O
learning	O
process	O
is	O
better	O
when	O
language	O
pairs	O
share	O
certain	O
characteristics	O
.	O
However	O
as	O
shown	O
in	O
sections	O
III	O
and	O
IV	O
of	O
Table	O
2	O
,	O
for	O
the	O
majority	O
of	O
the	O
language	O
pairs	O
,	O
specialised	O
multilingual	O
models	O
built	O
on	O
certain	O
domains	O
or	O
MT	O
types	O
do	O
not	O
perform	O
better	O
than	O
multilingual	O
models	O
which	O
contain	O
all	O
the	O
data	O
.	O
Section	O
IV	O
shows	O
the	O
results	O
of	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
methods	O
and	O
the	O
best	O
system	O
submitted	O
for	O
the	O
language	O
pair	O
in	O
that	O
competition	O
.	O
NR	O
implies	O
that	O
a	O
particular	O
result	O
was	O
not	O
reported	O
by	O
the	O
organisers	O
.	O
Zero	O
-	O
shot	O
results	O
are	O
coloured	O
in	O
grey	O
and	O
the	O
value	O
shows	O
the	O
difference	O
between	O
the	O
best	O
result	O
in	O
that	O
section	O
for	O
that	O
language	O
pair	O
and	O
itself	O
.	O

Multilingual	O
QE	O

The	O
values	O
displayed	O
diagonally	O
across	O
section	O
I	O
of	O
Table	O
2	O
show	O
the	O
results	O
for	O
supervised	O
,	O
bilingual	O
,	O
word	O
-	O
level	O
QE	O
models	O
where	O
the	O
model	O
was	O
trained	O
on	O
the	O
training	O
set	O
of	O
a	O
particular	O
language	O
pair	O
and	O
tested	O
on	O
the	O
test	O
set	O
of	O
the	O
same	O
language	O
pair	O
.	O
As	O
can	O
be	O
seen	O
in	O
section	O
V	O
,	O
the	O
architecture	O
outperforms	O
the	O
baselines	O
in	O
all	O
the	O
language	O
pairs	O
and	O
also	O
outperforms	O
the	O
majority	O
of	O
the	O
best	O
systems	O
from	O
previous	O
competitions	O
.	O
In	O
addition	O
to	O
the	O
target	O
word	O
F1	O
-	O
score	O
,	O
our	O
architecture	O
outperforms	O
the	O
baselines	O
and	O
best	O
systems	O
in	O
target	O
gaps	O
F1	O
-	O
score	O
and	O
source	O
words	O
F1	O
-	O
score	O
too	O
as	O
shown	O
in	O
Tables	O
5	O
and	O
6	O
.	O
In	O
the	O
following	O
sections	O
we	O
explore	O
its	O
behaviour	O
in	O
different	O
multilingual	O
settings	O
.	O

For	O
evaluation	O
,	O
we	O
used	O
the	O
approach	O
proposed	O
in	O
the	O
WMT	O
shared	O
tasks	O
in	O
which	O
the	O
classification	O
performance	O
is	O
calculated	O
using	O
the	O
multiplication	O
of	O
F1	O
-	O
scores	O
for	O
the	O
'	O
OK	O
'	O
and	O
'	O
BAD	O
'	O
classes	O
against	O
the	O
true	O
labels	O
independently	O
:	O
words	O
in	O
the	O
target	O
(	O
'	O
OK	O
'	O
for	O
correct	O
words	O
,	O
'	O
BAD	O
'	O
for	O
incorrect	O
words	O
)	O
,	O
gaps	O
in	O
the	O
target	O
(	O
'	O
OK	O
'	O
for	O
genuine	O
gaps	O
,	O
'	O
BAD	O
'	O
for	O
gaps	O
indicating	O
missing	O
words	O
)	O
and	O
source	O
words	O
(	O
'	O
BAD	O
'	O
for	O
words	O
that	O
lead	O
to	O
errors	O
in	O
the	O
target	O
,	O
'	O
OK	O
'	O
for	O
other	O
words	O
)	O
.	O
In	O
recent	O
WMT	O
shared	O
tasks	O
,	O
the	O
most	O
popular	O
category	O
was	O
predicting	O
quality	O
for	O
words	O
in	O
the	O
target	O
.	O
Therefore	O
,	O
in	O
Section	O
5	O
we	O
only	O
report	O
the	O
F1	O
-	O
score	O
for	O
words	O
in	O
the	O
target	O
.	O
Other	O
results	O
are	O
presented	O
in	O
the	O
supplementary	O
material	O
.	O
Prior	O
to	O
WMT	O
2019	O
,	O
organisers	O
provided	O
separate	O
scores	O
for	O
gaps	O
and	O
words	O
in	O
the	O
target	O
,	O
while	O
after	O
WMT	O
2019	O
they	O
produce	O
a	O
single	O
result	O
for	O
target	O
gaps	O
and	O
words	O
.	O
We	O
follow	O
this	O
latter	O
approach	O
.	O

Our	O
architecture	O
relies	O
on	O
the	O
XLM	O
-	O
R	O
transformer	O
model	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2020	O
)	O
to	O
derive	O
the	O
representations	O
of	O
the	O
input	O
sentences	O
.	O
XLM	O
-	O
R	O
has	O
been	O
trained	O
on	O
a	O
large	O
-	O
scale	O
multilingual	O
dataset	O
in	O
104	O
languages	O
,	O
totalling	O
2.5	O
TB	O
,	O
extracted	O
from	O
the	O
CommonCrawl	O
datasets	O
.	O
It	O
is	O
trained	O
using	O
only	O
RoBERTa	O
's	O
(	O
Liu	O
et	O
al	O
.	O
,	O
2019	O
)	O
masked	O
language	O
modelling	O
(	O
MLM	O
)	O
objective	O
.	O
XML	O
-	O
R	O
was	O
used	O
by	O
the	O
winning	O
systems	O
in	O
the	O
recent	O
WMT	O
2020	O
shared	O
task	O
on	O
sentence	O
-	O
level	O
QE	O
(	O
Ranasinghe	O
et	O
al	O
.	O
,	O
2020a;Lee	O
,	O
2020	O
;	O
.	O
This	O
motivated	O
us	O
to	O
use	O
a	O
similar	O
approach	O
for	O
wordlevel	O
QE	O
.	O

Our	O
architecture	O
adds	O
a	O
new	O
token	O
to	O
the	O
XLM	O
-	O
R	O
tokeniser	O
called	O
<	O
GAP	O
>	O
which	O
is	O
inserted	O
between	O
the	O
words	O
in	O
the	O
target	O
.	O
We	O
then	O
concatenate	O
the	O
source	O
and	O
the	O
target	O
with	O
a	O
[	O
SEP	O
]	O
token	O
and	O
we	O
feed	O
them	O
into	O
XLM	O
-	O
R.	O
A	O
simple	O
linear	O
layer	O
is	O
added	O
on	O
top	O
of	O
word	O
and	O
<	O
GAP	O
>	O
embeddings	O
to	O
predict	O
whether	O
it	O
is	O
"	O
Good	O
"	O
or	O
"	O
Bad	O
"	O
as	O
shown	O
in	O
Figure	O
1	O
.	O
The	O
training	O
configurations	O
and	O
the	O
system	O
specifications	O
are	O
presented	O
in	O
the	O
supplementary	O
material	O
.	O
We	O
used	O
several	O
language	O
pairs	O
for	O
which	O
word	O
-	O
level	O
QE	O
annotations	O
were	O
available	O
:	O
English	O
-	O
Chinese	O
(	O
En	O
-	O
Zh	O
)	O
,	O
English	O
-	O
Czech	O
(	O
En	O
-	O
Cs	O
)	O
,	O
English	O
-	O
German	O
(	O
En	O
-	O
De	O
)	O
,	O
English	O
-	O
Russian	O
(	O
En	O
-	O
Ru	O
)	O
,	O
English	O
-	O
Latvian	O
(	O
En	O
-	O
Lv	O
)	O
and	O
German	O
-	O
English	O
(	O
De	O
-	O
En	O
)	O
.	O
The	O
texts	O
are	O
from	O
a	O
variety	O
of	O
domains	O
and	O
the	O
translations	O
were	O
produced	O
using	O
both	O
neural	O
and	O
statistical	O
machine	O
translation	O
systems	O
.	O
More	O
details	O
about	O
these	O
datasets	O
can	O
be	O
found	O
in	O
Table	O
1	O
and	O
in	O
Fonseca	O
et	O
al	O
.	O
,	O
2019	O
;	O
.	O

Translating	O
Headers	O
of	O
Tabular	O
Data	O
:	O
A	O
Pilot	O
Study	O
of	O
Schema	O
Translation	O

Schema	O
translation	O
is	O
the	O
task	O
of	O
automatically	O
translating	O
headers	O
of	O
tabular	O
data	O
from	O
one	O
language	O
to	O
another	O
.	O
High	O
-	O
quality	O
schema	O
translation	O
plays	O
an	O
important	O
role	O
in	O
crosslingual	O
table	O
searching	O
,	O
understanding	O
and	O
analysis	O
.	O
Despite	O
its	O
importance	O
,	O
schema	O
translation	O
is	O
not	O
well	O
studied	O
in	O
the	O
community	O
,	O
and	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
machine	O
translation	O
models	O
can	O
not	O
work	O
well	O
on	O
this	O
task	O
because	O
of	O
two	O
intrinsic	O
differences	O
between	O
plain	O
text	O
and	O
tabular	O
data	O
:	O
morphological	O
difference	O
and	O
context	O
difference	O
.	O
To	O
facilitate	O
the	O
research	O
study	O
,	O
we	O
construct	O
the	O
first	O
parallel	O
dataset	O
for	O
schema	O
translation	O
,	O
which	O
consists	O
of	O
3,158	O
tables	O
with	O
11,979	O
headers	O
written	O
in	O
6	O
different	O
languages	O
,	O
including	O
English	O
,	O
Chinese	O
,	O
French	O
,	O
German	O
,	O
Spanish	O
,	O
and	O
Japanese	O
.	O
Also	O
,	O
we	O
propose	O
the	O
first	O
schema	O
translation	O
model	O
called	O
CAST	O
,	O
which	O
is	O
a	O
header	O
-	O
to	O
-	O
header	O
neural	O
machine	O
translation	O
model	O
augmented	O
with	O
schema	O
context	O
.	O
Specifically	O
,	O
we	O
model	O
a	O
target	O
header	O
and	O
its	O
context	O
as	O
a	O
directed	O
graph	O
to	O
represent	O
their	O
entity	O
types	O
and	O
relations	O
.	O
Then	O
CAST	O
encodes	O
the	O
graph	O
with	O
a	O
relational	O
-	O
aware	O
transformer	O
and	O
uses	O
another	O
transformer	O
to	O
decode	O
the	O
header	O
in	O
the	O
target	O
language	O
.	O
Experiments	O
on	O
our	O
dataset	O
demonstrate	O
that	O
CAST	O
significantly	O
outperforms	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
machine	O
translation	O
models	O
.	O
Our	O
dataset	O
will	O
be	O
released	O
at	O
https://github.com/microsoft/ContextualSP	O
.	O

Introduction	O

As	O
the	O
saying	O
goes	O
,	O
"	O
a	O
chart	O
is	O
worth	O
a	O
thousand	O
words	O
"	O
.	O
Nowadays	O
,	O
tremendous	O
amounts	O
of	O
tabular	O
data	O
written	O
in	O
various	O
languages	O
are	O
widely	O
used	O
in	O
Wikipedia	O
pages	O
,	O
research	O
papers	O
,	O
finance	O
reports	O
,	O
file	O
systems	O
,	O
and	O
databases	O
,	O
which	O
are	O
informative	O
.	O
Schema	O
translation	O
is	O
the	O
task	O
of	O
automatically	O
translating	O
headers	O
of	O
tabular	O
data	O
from	O
one	O
language	O
to	O
another	O
.	O
High	O
-	O
quality	O
schema	O
translation	O
plays	O
an	O
essential	O
role	O
in	O
cross	O
-	O
lingual	O
table	O
⇤	O
Work	O
done	O
during	O
an	O
internship	O
at	O
Microsoft	O
Research	O
.	O

No	O
.	O

Match	O
Hosted_by	O
Loc	O
.	O

Cost	O
(	O
$	O
)	O

Figure	O
1	O
:	O
An	O
illustrative	O
example	O
of	O
schema	O
translation	O
from	O
English	O
to	O
Chinese	O
.	O
1	O
-4	O
denotes	O
headers	O
with	O
abbreviation	O
,	O
polysemy	O
,	O
verb	O
-	O
object	O
phrase	O
and	O
special	O
symbol	O
,	O
respectively	O
.	O

searching	O
,	O
understanding	O
,	O
and	O
analysis	O
(	O
Zhang	O
and	O
Balog	O
,	O
2018;Deng	O
et	O
al	O
.	O
,	O
2019;Sherborne	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
Note	O
that	O
in	O
this	O
work	O
,	O
we	O
focus	O
on	O
translating	O
the	O
headers	O
instead	O
of	O
the	O
entire	O
table	O
content	O
,	O
since	O
for	O
each	O
entity	O
in	O
table	O
content	O
,	O
it	O
is	O
hard	O
to	O
decide	O
if	O
it	O
needs	O
to	O
be	O
translated	O
or	O
not	O
.	O
Over	O
translation	O
could	O
even	O
have	O
negative	O
effects	O
in	O
reality	O
.	O
Despite	O
its	O
importance	O
,	O
most	O
research	O
efforts	O
are	O
dedicated	O
to	O
plain	O
text	O
machine	O
translation	O
(	O
Sutskever	O
et	O
al	O
.	O
,	O
2014;Bahdanau	O
et	O
al	O
.	O
,	O
2015;Vaswani	O
et	O
al	O
.	O
,	O
2017;Yang	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
and	O
schema	O
translation	O
is	O
not	O
well	O
studied	O
in	O
the	O
community	O
,	O
to	O
the	O
best	O
of	O
our	O
knowledge	O
.	O
According	O
to	O
our	O
preliminary	O
study	O
,	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
machine	O
translation	O
(	O
NMT	O
)	O
systems	O
can	O
not	O
work	O
well	O
on	O
schema	O
translation	O
because	O
of	O
two	O
intrinsic	O
differences	O
between	O
plain	O
text	O
and	O
tabular	O
data	O
:	O
morphological	O
difference	O
and	O
context	O
difference	O
.	O

Morphological	O
Difference	O
.	O
The	O
morphology	O
of	O
table	O
headers	O
differs	O
from	O
that	O
of	O
plain	O
text	O
in	O
the	O
following	O
four	O
aspects	O
.	O
First	O
,	O
headers	O
are	O
always	O
phrases	O
and	O
they	O
usually	O
contain	O
a	O
lot	O
of	O
domainspecific	O
abbreviations	O
(	O
e.g.	O
,	O
as	O
shown	O
in	O
Figure	O
1	O
,	O
"	O
No	O
.	O
"	O
is	O
the	O
abbreviation	O
of	O
"	O
Number	O
"	O
and	O
the	O
"	O
Loc	O
.	O
"	O
is	O
short	O
for	O
"	O
Location	O
"	O
)	O
and	O
special	O
symbols	O
(	O
e.g.	O
,	O
"	O
$	O
"	O
means	O
"	O
dollar	O
"	O
in	O
Figure	O
1	O
)	O
.	O
Second	O
,	O
verb	O
-	O
object	O
phrases	O
are	O
frequently	O
used	O
as	O
headers	O
which	O
indicate	O
a	O
subject	O
-	O
object	O
relationship	O
between	O
two	O
columns	O
.	O
For	O
example	O
,	O
"	O
Hosted	O
by	O
"	O
in	O
Figure	O
1	O
indicates	O
a	O
host	O
relationship	O
between	O
the	O
second	O
and	O
the	O
third	O
columns	O
.	O
Third	O
,	O
special	O
tokenizations	O
like	O
CamelCase	O
and	O
underscore	O
are	O
idiomatic	O
usages	O
in	O
headers	O
.	O
At	O
last	O
,	O
capitalized	O
words	O
are	O
particularly	O
preferred	O
in	O
order	O
to	O
capture	O
more	O
readers	O
'	O
attention	O
for	O
headers	O
.	O
These	O
special	O
word	O
-	O
forms	O
are	O
commonly	O
used	O
in	O
headers	O
but	O
rarely	O
seen	O
in	O
plain	O
text	O
.	O
Therefore	O
,	O
the	O
NMT	O
models	O
trained	O
with	O
a	O
massive	O
amount	O
of	O
plain	O
text	O
can	O
not	O
be	O
directly	O
applied	O
to	O
schema	O
translation	O
.	O

Context	O
Difference	O
.	O
Compared	O
with	O
plain	O
text	O
,	O
which	O
is	O
a	O
sequence	O
of	O
words	O
,	O
tables	O
have	O
welldefined	O
structures	O
,	O
and	O
understanding	O
a	O
table	O
's	O
structure	O
is	O
crucial	O
for	O
schema	O
translation	O
.	O
Specifically	O
,	O
a	O
table	O
consists	O
of	O
an	O
ordered	O
arrangement	O
of	O
rows	O
and	O
columns	O
.	O
Each	O
column	O
header	O
describes	O
the	O
concept	O
of	O
that	O
column	O
.	O
The	O
intersection	O
of	O
a	O
row	O
and	O
a	O
column	O
is	O
called	O
a	O
cell	O
.	O
Each	O
cell	O
contains	O
entities	O
of	O
the	O
column	O
header	O
it	O
belongs	O
to	O
.	O
This	O
structure	O
plays	O
an	O
important	O
role	O
in	O
schema	O
translation	O
,	O
especially	O
for	O
polysemy	O
words	O
and	O
abbreviation	O
words	O
.	O
For	O
example	O
,	O
in	O
Figure	O
1	O
,	O
the	O
header	O
"	O
Match	O
"	O
could	O
be	O
translated	O
to	O
"	O
kÙ	O
(	O
Matchstick	O
)	O
"	O
,	O
"	O
9	O
M	O
(	O
Mapping	O
)	O
"	O
,	O
and	O
"	O
'	O
[	O
(	O
Competition	O
)	O
"	O
,	O
but	O
its	O
sibling	O
column	O
header	O
"	O
Hosted_by	O
"	O
provides	O
important	O
clues	O
that	O
the	O
table	O
might	O
belong	O
to	O
the	O
domain	O
of	O
sport	O
.	O
Thus	O
,	O
translating	O
"	O
Match	O
"	O
to	O
"	O
'	O
[	O
(	O
Competition	O
)	O
"	O
is	O
more	O
appropriate	O
in	O
the	O
context	O
.	O
Moreover	O
,	O
a	O
column	O
header	O
's	O
cell	O
values	O
could	O
also	O
provide	O
hints	O
to	O
infer	O
the	O
meaning	O
of	O
the	O
header	O
.	O
For	O
example	O
,	O
successive	O
numerical	O
cell	O
values	O
indicate	O
that	O
"	O
No	O
.	O
"	O
might	O
be	O
an	O
identity	O
column	O
in	O
Figure	O
1	O
.	O
NMT	O
models	O
trained	O
with	O
plain	O
text	O
have	O
never	O
seen	O
the	O
structure	O
of	O
tables	O
,	O
and	O
consequently	O
,	O
they	O
perform	O
poorly	O
in	O
schema	O
translation	O
.	O

Although	O
the	O
context	O
information	O
of	O
tables	O
is	O
important	O
,	O
how	O
to	O
effectively	O
use	O
it	O
for	O
schema	O
translation	O
is	O
challenging	O
.	O
On	O
the	O
one	O
hand	O
,	O
the	O
NMT	O
model	O
needs	O
to	O
make	O
use	O
of	O
the	O
context	O
information	O
to	O
make	O
word	O
-	O
sense	O
disambiguation	O
for	O
polysemy	O
headers	O
and	O
abbreviation	O
headers	O
.	O
For	O
another	O
,	O
the	O
context	O
information	O
should	O
not	O
bring	O
additional	O
noise	O
when	O
translating	O
the	O
target	O
header	O
.	O

To	O
facilitate	O
the	O
research	O
study	O
,	O
we	O
construct	O
the	O
first	O
parallel	O
dataset	O
for	O
schema	O
translation	O
written	O
in	O
six	O
different	O
languages	O
.	O
It	O
consists	O
of	O
3,158	O
tables	O
with	O
11,979	O
headers	O
written	O
in	O
six	O
differ	O
-	O
ent	O
languages	O
,	O
including	O
English	O
,	O
Chinese	O
,	O
French	O
,	O
German	O
,	O
Spanish	O
,	O
and	O
Japanese	O
.	O

Furthermore	O
,	O
to	O
address	O
the	O
challenges	O
in	O
schema	O
translation	O
,	O
we	O
propose	O
a	O
Context	O
Aware	O
Schema	O
Translation	O
(	O
CAST	O
)	O
model	O
,	O
which	O
is	O
a	O
header	O
-	O
to	O
-	O
header	O
neural	O
machine	O
translation	O
model	O
augmented	O
with	O
table	O
context	O
.	O
Specifically	O
,	O
we	O
model	O
a	O
target	O
header	O
and	O
its	O
context	O
as	O
a	O
directed	O
graph	O
to	O
represent	O
their	O
entity	O
types	O
and	O
structural	O
relations	O
.	O
Then	O
CAST	O
encodes	O
the	O
graph	O
with	O
a	O
relational	O
-	O
aware	O
transformer	O
and	O
uses	O
another	O
transformer	O
to	O
decode	O
the	O
header	O
in	O
the	O
target	O
language	O
.	O
The	O
advantages	O
of	O
our	O
approach	O
come	O
from	O
two	O
folds	O
:	O
(	O
1	O
)	O
The	O
structure	O
relationships	O
make	O
the	O
transformer	O
encoder	O
capture	O
the	O
structural	O
information	O
and	O
learn	O
a	O
contextualized	O
representation	O
for	O
the	O
target	O
header	O
;	O
(	O
2	O
)	O
The	O
entity	O
types	O
differentiate	O
the	O
target	O
header	O
from	O
its	O
context	O
and	O
thus	O
help	O
denoise	O
the	O
target	O
header	O
translation	O
.	O

Experiments	O
on	O
our	O
dataset	O
demonstrate	O
that	O
CAST	O
significantly	O
outperforms	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
machine	O
translation	O
models	O
.	O
Our	O
contributions	O
are	O
summarized	O
as	O
follows	O
.	O

•	O
We	O
propose	O
the	O
task	O
of	O
schema	O
translation	O
,	O
and	O
discuss	O
its	O
differences	O
with	O
a	O
plain	O
text	O
translation	O
.	O
To	O
facilitate	O
the	O
research	O
study	O
,	O
we	O
construct	O
the	O
first	O
parallel	O
schema	O
translation	O
dataset	O
.	O

•	O
We	O
propose	O
a	O
header	O
-	O
to	O
-	O
header	O
context	O
-	O
aware	O
schema	O
translation	O
model	O
,	O
called	O
CAST	O
,	O
for	O
the	O
new	O
schema	O
translation	O
task	O
.	O
Specifically	O
,	O
we	O
use	O
the	O
transformer	O
self	O
-	O
attention	O
mechanism	O
to	O
encode	O
the	O
schema	O
over	O
predefined	O
entity	O
types	O
and	O
structural	O
relationships	O
,	O
making	O
it	O
aware	O
of	O
the	O
schema	O
context	O
.	O

•	O
Experiments	O
on	O
our	O
proposed	O
dataset	O
demonstrate	O
that	O
our	O
approach	O
significantly	O
outperforms	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
machine	O
translation	O
models	O
in	O
schema	O
translation	O
.	O

Schema	O
Translation	O
Dataset	O

To	O
address	O
the	O
need	O
for	O
a	O
dataset	O
for	O
the	O
new	O
schema	O
translation	O
task	O
,	O
we	O
construct	O
the	O
first	O
parallel	O
schema	O
translation	O
dataset	O
.	O
It	O
consists	O
of	O
3,158	O
tables	O
with	O
11,979	O
headers	O
written	O
in	O
six	O
different	O
languages	O
,	O
including	O
English	O
,	O
Chinese	O
,	O
French	O
,	O
German	O
,	O
Spanish	O
,	O
and	O
Japanese	O
.	O
In	O
this	O
section	O
,	O
we	O
will	O
first	O
introduce	O
our	O
construction	O
methodology	O
and	O
then	O
analyze	O
the	O
characteristics	O
of	O
our	O
dataset	O
.	O

58	O

Dataset	O
Construction	O

We	O
construct	O
the	O
dataset	O
in	O
two	O
steps	O
:	O
collecting	O
3,158	O
English	O
tables	O
and	O
then	O
manually	O
translating	O
the	O
schema	O
of	O
English	O
tables	O
to	O
other	O
languages	O
.	O
(	O
Pasupat	O
and	O
Liang	O
,	O
2015	O
)	O
,	O
in	O
which	O
they	O
randomly	O
select	O
2,108	O
multidomain	O
data	O
tables	O
in	O
English	O
from	O
Wikipedia	O
with	O
at	O
least	O
eight	O
rows	O
and	O
five	O
columns	O
.	O
Secondly	O
,	O
we	O
manually	O
collect	O
176	O
English	O
tables	O
from	O
the	O
search	O
engine	O
covering	O
multiple	O
domains	O
like	O
retail	O
,	O
education	O
,	O
and	O
government	O
.	O
At	O
last	O
,	O
we	O
select	O
all	O
the	O
tables	O
that	O
appear	O
in	O
the	O
training	O
set	O
and	O
development	O
set	O
from	O
the	O
Spider	O
dataset	O
(	O
Yu	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
which	O
contains	O
200	O
databases	O
covering	O
138	O
different	O
domains	O
.	O
Finally	O
,	O
we	O
obtained	O
3,158	O
tables	O
with	O
11,979	O
headers	O
in	O
total	O
.	O

Context	O
Aware	O
Schema	O
Annotation	O
.	O
To	O
reduce	O
the	O
translation	O
effort	O
,	O
we	O
first	O
use	O
Google	O
translator	O
1	O
to	O
automatically	O
translate	O
the	O
English	O
headers	O
to	O
five	O
target	O
languages	O
,	O
header	O
by	O
header	O
.	O
Then	O
based	O
on	O
the	O
Google	O
translations	O
,	O
we	O
recruit	O
three	O
professional	O
translators	O
for	O
each	O
language	O
to	O
manually	O
check	O
and	O
modify	O
the	O
translations	O
if	O
inappropriate	O
.	O

In	O
this	O
process	O
,	O
we	O
found	O
that	O
Google	O
translator	O
is	O
not	O
good	O
enough	O
in	O
schema	O
translation	O
since	O
industry	O
jargon	O
and	O
abbreviations	O
are	O
commonly	O
used	O
in	O
column	O
headers	O
.	O
Table	O
1	O
shows	O
some	O
example	O
headers	O
and	O
their	O
paraphrases	O
under	O
different	O
domains	O
in	O
our	O
dataset	O
.	O
However	O
,	O
domain	O
information	O
is	O
implicit	O
,	O
and	O
the	O
meaning	O
of	O
the	O
header	O
needs	O
to	O
be	O
inferred	O
carefully	O
from	O
the	O
entire	O
table	O
context	O
.	O
To	O
get	O
more	O
precise	O
translations	O
,	O
we	O
provide	O
three	O
kinds	O
of	O
additional	O
information	O
as	O
a	O
schema	O
context	O
:	O
(	O
1	O
)	O
a	O
whole	O
table	O
with	O
structural	O
information	O
,	O
including	O
its	O
table	O
name	O
,	O
column	O
headers	O
and	O
cell	O
values	O
;	O
(	O
2	O
)	O
an	O
original	O
web	O
-	O
page	O
URL	O
for	O
the	O
table	O
from	O
the	O
Wikipedia	O
website	O
;	O
(	O
3	O
)	O
some	O
natural	O
language	O
question	O
/	O
answer	O
pairs	O
about	O
the	O
table	O
2	O
.	O
Our	O
translators	O
are	O
asked	O
to	O
first	O
understand	O
the	O
context	O
of	O
the	O
given	O
schema	O
before	O
validating	O
the	O
translations	O
.	O
We	O
find	O
that	O
the	O
modification	O
rate	O
is	O
40	O
%	O
,	O
which	O
indicates	O
that	O
the	O
provided	O
context	O
is	O
very	O
useful	O
.	O
Finally	O
,	O
we	O
further	O
verify	O
the	O
annotated	O
data	O
by	O
asking	O
a	O
different	O
translator	O
to	O
check	O
if	O
the	O
headers	O
are	O
correctly	O
translated	O
.	O

Data	O
Statistics	O
and	O
Analysis	O

As	O
we	O
know	O
,	O
the	O
translation	O
cost	O
is	O
expensive	O
,	O
and	O
we	O
provide	O
parallel	O
corpus	O
in	O
six	O
languages	O
,	O
which	O
limits	O
the	O
volume	O
of	O
translated	O
headers	O
.	O
On	O
the	O
basis	O
of	O
our	O
statistics	O
,	O
the	O
average	O
validating	O
speed	O
is	O
100	O
headers	O
/	O
hour	O
and	O
we	O
spend	O
159.34	O
⇤	O
5	O
hours	O
in	O
total	O
.	O
This	O
speed	O
is	O
much	O
slower	O
than	O
the	O
plain	O
text	O
translation	O
since	O
our	O
translators	O
need	O
to	O
read	O
large	O
amounts	O
of	O
different	O
domain	O
-	O
specific	O
contexts	O
to	O
help	O
disambiguation	O
.	O
To	O
this	O
end	O
,	O
we	O
make	O
our	O
best	O
effort	O
and	O
translate	O
11,979	O
headers	O
,	O
spending	O
6,625	O
USD	O
in	O
total	O
.	O
According	O
to	O
our	O
translators	O
'	O
feedback	O
,	O
the	O
context	O
is	O
quite	O
helpful	O
in	O
understanding	O
the	O
meaning	O
of	O
headers	O
.	O
We	O
will	O
also	O
release	O
these	O
contexts	O
together	O
with	O
our	O
schema	O
translation	O
dataset	O
to	O
facilitate	O
further	O
study	O
.	O

Dataset	O
Analysis	O
.	O
To	O
have	O
a	O
more	O
quantitative	O
analysis	O
of	O
our	O
dataset	O
,	O
we	O
count	O
the	O
ratio	O
of	O
headers	O
containing	O
four	O
lexical	O
features	O
,	O
including	O
abbreviation	O
,	O
symbol	O
characters	O
,	O
verb	O
-	O
object	O
phrase	O
and	O
capitalized	O
character	O
.	O
As	O
we	O
can	O
see	O
in	O
table	O
2	O
,	O
these	O
lexical	O
features	O
commonly	O
occur	O
in	O
headers	O
,	O
making	O
them	O
quite	O
different	O
from	O
plain	O
text	O
.	O

To	O
help	O
better	O
understand	O
the	O
domains	O
of	O
the	O
collected	O
tables	O
,	O
we	O
firstly	O
use	O
a	O
44	O
-	O
category	O
ontology	O
presented	O
in	O
Wikipedia	O
:	O
WikiProject	O
Council	O
/	O
Directory	O
as	O
our	O
domain	O
category	O
.	O
Then	O
we	O
randomly	O
sample	O
500	O
tables	O
in	O
the	O
training	O
set	O
and	O
manually	O
label	O
the	O
domains	O
.	O
According	O
to	O
our	O
statistics	O
,	O
our	O
dataset	O
covers	O
all	O
44	O
domains	O
.	O
In	O
detail	O
,	O
the	O
Sports	O
,	O
Countries	O
,	O
Economics	O
,	O
and	O
Music	O
topics	O
together	O
comprise	O
44.6	O
%	O
of	O
our	O
dataset	O
,	O
but	O
the	O
other	O
55.4	O
%	O
is	O
composed	O
of	O
broader	O
topics	O
such	O
as	O
Business	O
,	O
Education	O
,	O
Science	O
,	O
and	O
Government	O
.	O

Methodology	O

In	O
this	O
section	O
,	O
we	O
describe	O
our	O
schema	O
translation	O
approach	O
in	O
detail	O
.	O
We	O
first	O
introduce	O
the	O
requirement	O
and	O
our	O
definition	O
for	O
the	O
schema	O
translation	O
task	O
and	O
then	O
introduce	O
the	O
model	O
architecture	O
.	O

Task	O
Requirement	O

In	O
schema	O
translation	O
,	O
both	O
the	O
meaning	O
of	O
the	O
headers	O
and	O
the	O
structural	O
information	O
like	O
order	O
and	O
numbers	O
must	O
be	O
completely	O
transferred	O
to	O
the	O
target	O
language	O
.	O
Obviously	O
,	O
this	O
requirement	O
can	O
not	O
be	O
met	O
by	O
translating	O
schema	O
as	O
a	O
whole	O
with	O
the	O
traditional	O
sequence	O
-	O
to	O
-	O
sequence	O
NMT	O
models	O
because	O
it	O
can	O
not	O
achieve	O
precisely	O
token	O
level	O
alignment	O
.	O
For	O
example	O
,	O
when	O
concatenating	O
all	O
headers	O
with	O
a	O
separator	O
"	O
|	O
"	O
,	O
the	O
separator	O
can	O
be	O
easily	O
lost	O
during	O
translation	O
.	O
To	O
meet	O
this	O
requirement	O
,	O
we	O
employ	O
a	O
header	O
-	O
to	O
-	O
header	O
translation	O
manner	O
in	O
this	O
work	O
,	O
which	O
translates	O
one	O
header	O
at	O
a	O
time	O
.	O

Task	O
Definition	O

We	O
define	O
a	O
column	O
header	O
as	O
H	O
i	O
=	O
hh	O
1	O
,	O
.	O
.	O
.	O
,	O
h	O
n	O
i	O
,	O
where	O
h	O
j	O
is	O
the	O
jth	O
token	O
of	O
the	O
header	O
in	O
the	O
source	O
language	O
.	O

Let	O
C	O
i	O
=	O
(	O
S	O
i	O
,	O
V	O
i	O
)	O
denote	O
the	O
context	O
of	O
H	O
i	O
.	O

It	O
is	O
made	O
up	O
of	O
a	O
set	O
of	O
selected	O
cell	O
values	O
V	O
i	O
=	O
{	O
v	O
1	O
,	O
.	O
.	O
.	O
,	O
v	O
t	O
}	O
of	O
H	O
i	O
and	O
the	O
rest	O
of	O
headers	O

S	O
i	O
=	O
[	O
H	O
1	O
,	O
.	O
.	O
.	O
,	O
H	O
i	O
1	O
,	O
H	O
i+1	O
,	O
.	O
.	O
.	O
,	O
H	O
m	O
]	O

in	O
the	O
schema	O
.	O
The	O
translation	O
of	O
H	O
i	O
is	O
denoted	O
as	O
Y	O
i	O
=	O
hy	O
1	O
,	O
.	O
.	O
.	O
,	O
y	O
m	O
i	O
,	O
where	O
y	O
j	O
is	O
the	O
jth	O
token	O
of	O
the	O
header	O
in	O
the	O
target	O
language	O
.	O
Taking	O
a	O
header	O
H	O
and	O
its	O
corresponding	O
context	O
C	O
as	O
input	O
,	O
the	O
model	O
outputs	O
the	O
header	O
Y	O
in	O
the	O
target	O
language	O
.	O

Model	O

Basically	O
,	O
our	O
model	O
adopts	O
a	O
Transformer	O
encoderdecoder	O
architecture	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
,	O
which	O
takes	O
the	O
source	O
language	O
header	O
with	O
its	O
corresponding	O
context	O
as	O
inputs	O
and	O
generates	O
the	O
translation	O
for	O
the	O
target	O
language	O
header	O
as	O
outputs	O
.	O
Specifically	O
,	O
we	O
model	O
the	O
target	O
header	O
and	O
its	O
context	O
as	O
a	O
directed	O
graph	O
and	O
use	O
the	O
transformer	O
self	O
-	O
attention	O
to	O
encode	O
them	O
over	O
two	O
predefined	O
structural	O
relationships	O
and	O
three	O
entity	O
types	O
.	O
Figure	O
2	O
depicts	O
the	O
overall	O
architecture	O
of	O
our	O
model	O
via	O
an	O
illustrative	O
example	O
.	O

Relation	O
-	O
Aware	O
Self	O
-	O
Attention	O
.	O
First	O
,	O
we	O
introduce	O
self	O
-	O
attention	O
and	O
then	O
its	O
extension	O
,	O
relationaware	O
self	O
-	O
attention	O
.	O
Consider	O
a	O
sequence	O
of	O
inputs	O

X	O
=	O
{	O
x	O
i	O
}	O
n	O
i=1	O

where	O
x	O
i	O
2	O
R	O
dx	O
.	O
Self	O
-	O
attention	O
introduced	O
by	O
Vaswani	O
et	O
al	O
.	O
(	O
2017	O
)	O
transforms	O
each	O
x	O
i	O
into	O
z	O
i	O
2	O
R	O
dx	O
as	O
follows	O
:	O

e	O
ij	O
=	O
x	O
i	O
W	O
Q	O
(	O
x	O
j	O
W	O
K	O
)	O
T	O
p	O
d	O
z	O
↵	O
ij	O
=	O
softmax	O
j	O
{	O
e	O
ij	O
}	O
(	O
1	O
)	O
z	O
i	O
=	O
n	O
X	O
j=1	O
↵	O
ij	O
(	O
x	O
j	O
W	O
V	O
)	O

where	O
dz	O
)	O
.	O
Shaw	O
et	O
al	O
.	O
(	O
2018	O
)	O
proposes	O
an	O
extension	O
to	O
selfattention	O
to	O
consider	O
the	O
pairwise	O
relationships	O
between	O
input	O
tokens	O
by	O
changing	O
Equation	O
(	O
1	O
)	O
as	O
follows	O
:	O

W	O
Q	O
,	O
W	O
K	O
,	O
W	O
V	O
2	O
R	O
dx	O
⇥	O
(	O

e	O
ij	O
=	O
x	O
i	O
W	O
Q	O
(	O
x	O
j	O
W	O
K	O
+	O
r	O
K	O
ij	O
)	O
)	O
T	O
p	O
d	O
z	O
z	O
i	O
=	O
n	O
X	O
j=1	O
↵	O
ij	O
(	O
x	O
j	O
W	O
V	O
+	O
r	O
V	O
ij	O
)	O
(	O
2	O
)	O

Here	O
the	O
r	O
ij	O
terms	O
encode	O
the	O
known	O
relationships	O
between	O
the	O
two	O
tokens	O
x	O
i	O
and	O
x	O
j	O
in	O
the	O
input	O
sequence	O
.	O
In	O
this	O
way	O
,	O
this	O
self	O
-	O
attention	O
is	O
biased	O
toward	O
some	O
pre	O
-	O
defined	O
relationships	O
using	O
the	O
relation	O
vector	O
r	O
ij	O
in	O
each	O
layer	O
when	O
learning	O
the	O
contextualized	O
embedding	O
.	O
Specifically	O
,	O
they	O
use	O
it	O
to	O
represent	O
the	O
relative	O
position	O
information	O
between	O
sequence	O
elements	O
.	O
More	O
details	O
could	O
be	O
found	O
in	O
their	O
work	O
(	O
Shaw	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
Figure	O
2	O
:	O
An	O
overview	O
of	O
CAST	O
with	O
an	O
illustrative	O
example	O
of	O
English	O
-	O
to	O
-	O
Chinese	O
schema	O
translation	O
.	O
Firstly	O
,	O
the	O
target	O
header	O
"	O
Chinese	O
"	O
and	O
its	O
context	O
are	O
modeled	O
as	O
a	O
directed	O
graph	O
.	O
Then	O
a	O
stack	O
of	O
relation	O
-	O
aware	O
transformers	O
encodes	O
the	O
input	O
sequence	O
X	O
to	O
X	O
0	O
with	O
a	O
relational	O
matrix	O
R	O
induced	O
from	O
the	O
graph	O
.	O

Inspired	O
by	O
Shaw	O
et	O
al	O
.	O
(	O
2018	O
)	O
,	O
we	O
model	O
the	O
target	O
header	O
and	O
its	O
context	O
as	O
a	O
labeled	O
directed	O
graph	O
and	O
use	O
the	O
same	O
formulation	O
of	O
relationaware	O
self	O
-	O
attention	O
as	O
Shaw	O
et	O
al	O
.	O
(	O
2018	O
)	O
.	O
Here	O

X	O
=	O
{	O
x	O
i	O
}	O
n	O

i=1	O
are	O
initial	O
embeddings	O
of	O
our	O
input	O
sequence	O
,	O
and	O
the	O
relational	O
matrix	O
R	O
is	O
induced	O
from	O
the	O
input	O
graph	O
,	O
where	O
r	O
ij	O
is	O
a	O
learned	O
embedding	O
according	O
to	O
the	O
type	O
of	O
edge	O
that	O
x	O
i	O
and	O
x	O
j	O
hold	O
in	O
the	O
directed	O
input	O
graph	O
.	O
The	O
following	O
section	O
will	O
describe	O
the	O
set	O
of	O
relations	O
our	O
model	O
uses	O
to	O
encode	O
a	O
target	O
header	O
concatenated	O
with	O
its	O
context	O
.	O

Input	O
Graph	O
.	O
We	O
model	O
a	O
target	O
header	O
and	O
its	O
context	O
as	O
a	O
directed	O
graph	O
to	O
represent	O
their	O
entity	O
types	O
and	O
structural	O
relations	O
.	O
Firstly	O
,	O
we	O
induce	O
two	O
kinds	O
of	O
edges	O
to	O
denote	O
the	O
structural	O
relationships	O
between	O
the	O
target	O
header	O
and	O
its	O
context	O
:	O
sibling	O
header	O
(	O
i.e.	O
,	O
an	O
edge	O
point	O
from	O
tokens	O
in	O
S	O
to	O
tokens	O
in	O
the	O
target	O
header	O
.	O
)	O
,	O
and	O
belonging	O
value	O
(	O
i.e.	O
,	O
an	O
edge	O
point	O
from	O
tokens	O
in	O
V	O
to	O
tokens	O
in	O
the	O
target	O
header	O
.	O
)	O
.	O
In	O
this	O
sense	O
,	O
it	O
could	O
incorporate	O
the	O
structural	O
information	O
into	O
the	O
contextualized	O
representation	O
of	O
the	O
target	O
header	O
.	O

Then	O
,	O
we	O
define	O
three	O
sorts	O
of	O
entity	O
types	O
to	O
distinguish	O
the	O
target	O
header	O
from	O
its	O
context	O
.	O
Specifically	O
,	O
for	O
a	O
token	O
in	O
the	O
target	O
header	O
,	O
we	O
assign	O
a	O
special	O
edge	O
Target	O
point	O
to	O
itself	O
,	O
denoting	O
the	O
entity	O
type	O
.	O
For	O
tokens	O
in	O
S	O
and	O
V	O
,	O
we	O
assign	O
them	O
different	O
edges	O
point	O
to	O
themselves	O
,	O
e.g.	O
,	O
Header	O
,	O
and	O
Value	O
respectively	O
.	O
Figure	O
2	O
illustrates	O
an	O
example	O
graph	O
(	O
with	O
actual	O
edges	O
and	O
labels	O
)	O
and	O
its	O
induced	O
relational	O
matrix	O
R.	O
Initial	O
Token	O
Embedding	O
.	O
We	O
obtain	O
the	O
initial	O
token	O
embedding	O
by	O
a	O
pre	O
-	O
trained	O
transformer	O
encoder	O
before	O
feeding	O
it	O
to	O
the	O
ration	O
-	O
aware	O
transformer	O
.	O
To	O
obtain	O
the	O
input	O
sequence	O
,	O
each	O
element	O
in	O
S	O
and	O
V	O
are	O
firstly	O
concatenated	O
with	O
a	O
vertical	O
bar	O
"	O
|	O
"	O
.	O
Then	O
,	O
the	O
target	O
header	O
H	O
,	O
the	O
rest	O
of	O
the	O
headers	O
S	O
,	O
and	O
the	O
selected	O
cell	O
values	O
V	O
are	O
concatenated	O
by	O
a	O
separator	O
symbol	O
"	O
[	O
sep	O
]	O
"	O
.	O
At	O
last	O
,	O
following	O
,	O
an	O
additional	O
source	O
language	O
token	O
"	O
hsrci	O
"	O
is	O
added	O
at	O
the	O
front	O
to	O
help	O
the	O
pretrained	O
model	O
identify	O
the	O
source	O
language	O
.	O
The	O
encoder	O
then	O
transforms	O
the	O
final	O
input	O
sequence	O
into	O
a	O
sequence	O
of	O
embedding	O

X	O
=	O
[	O
x	O
1	O
,	O
.	O
.	O
.	O
,	O
x	O
l	O
]	O
.	O

Then	O
we	O
feed	O
them	O
to	O
the	O
relational	O
aware	O
layers	O
and	O
get	O
the	O
final	O
contextualized	O
sequence	O
of	O
embedding	O
X	O

0	O
=	O
[	O
x	O
0	O
1	O
,	O
.	O
.	O
.	O
,	O
x	O
0	O
l	O
]	O
.	O

Decoder	O
.	O
The	O
goal	O
of	O
the	O
decoder	O
is	O
to	O
autoregressively	O
generate	O
the	O
translated	O
column	O
header	O
Y	O
=	O
hy	O
1	O
,	O
.	O
.	O
.	O
,	O
y	O
m	O
i.	O
Specifically	O
,	O
taking	O
X	O
0	O
and	O
the	O
representation	O
of	O
previously	O
output	O
token	O
as	O
input	O
,	O
the	O
decoder	O
predicts	O
the	O
translation	O
token	O
by	O
token	O
until	O
an	O
ending	O
signal	O
hendi	O
is	O
generated	O
.	O
Similar	O
to	O
the	O
encoder	O
,	O
a	O
special	O
token	O
htgti	O
which	O
indicates	O
the	O
target	O
language	O
is	O
added	O
at	O
the	O
front	O
to	O
guide	O
the	O
prediction	O
of	O
the	O
target	O
language	O
.	O

Experiments	O

In	O
this	O
section	O
,	O
we	O
conduct	O
experiments	O
on	O
our	O
proposed	O
schema	O
translation	O
dataset	O
to	O
evaluate	O
the	O
effectiveness	O
of	O
our	O
approach	O
.	O
Furthermore	O
,	O
we	O
ablate	O
different	O
ways	O
of	O
context	O
modeling	O
in	O
our	O
approach	O
to	O
understand	O
their	O
contributions	O
.	O
At	O
last	O
,	O
we	O
conduct	O
a	O
qualitative	O
analysis	O
and	O
show	O
example	O
cases	O
and	O
their	O
predicting	O
results	O
.	O

Experiment	O
Setup	O

Baseline	O
.	O
We	O
choose	O
two	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
NMT	O
models	O
,	O
including	O
M2M-100	O
and	O
MBart-50M2	O
M	O
(	O
Tang	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
as	O
our	O
baselines	O
.	O
Specifically	O
,	O
both	O
of	O
the	O
baseline	O
models	O
employ	O
the	O
Transformer	O
sequence	O
-	O
to	O
-	O
sequence	O
architecture	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
to	O
capture	O
features	O
from	O
source	O
language	O
input	O
and	O
generate	O
the	O
translation	O
.	O
The	O
M2M-100	O
is	O
directly	O
trained	O
on	O
large	O
-	O
scaled	O
translation	O
data	O
while	O
MBart-50M2	O
M	O
is	O
firstly	O
pre	O
-	O
trained	O
with	O
a	O
"	O
Multilingual	O
Denoising	O
Pretraining	O
"	O
objective	O
and	O
then	O
fine	O
-	O
tuned	O
in	O
machine	O
-	O
translation	O
task	O
.	O
We	O
evaluate	O
the	O
baseline	O
models	O
with	O
the	O
following	O
settings	O
:	O

•	O
Base	O
:	O
The	O
original	O
NMT	O
models	O
without	O
finetuning	O
on	O
the	O
schema	O
dataset	O
.	O
•	O
H2H	O
:	O
The	O
NMT	O
models	O
that	O
are	O
fine	O
-	O
tuned	O
on	O
our	O
schema	O
translation	O
dataset	O
in	O
a	O
headerto	O
-	O
header	O
manner	O
.	O
•	O
H2H+CXT	O
:	O
The	O
NMT	O
models	O
are	O
fine	O
-	O
tuned	O
by	O
concatenating	O
a	O
target	O
header	O
and	O
its	O
context	O
as	O
input	O
and	O
translating	O
the	O
target	O
header	O
.	O
•	O
H2H+CXT+ExtL	O
:	O
The	O
NMT	O
models	O
with	O
two	O
extra	O
Transformers	O
layers	O
at	O
the	O
end	O
of	O
the	O
encoder	O
,	O
and	O
are	O
fine	O
-	O
tuned	O
with	O
the	O
same	O
setting	O
as	O
H2H+CXT	O
.	O

Besides	O
NMT	O
models	O
,	O
we	O
also	O
trained	O
a	O
phrasebased	O
statistical	O
machine	O
translation	O
(	O
PB	O
-	O
SMT	O
)	O
schema	O
translation	O
model	O
with	O
Moses	O
3	O
(	O
Koehn	O
et	O
al	O
.	O
,	O
2007	O
)	O
,	O
with	O
the	O
same	O
data	O
split	O
.	O

Evaluation	O
Metrics	O
.	O
We	O
evaluate	O
the	O
performances	O
of	O
different	O
models	O
with	O
the	O
4	O
-	O
gram	O
BLEU	O
(	O
Papineni	O
et	O
al	O
.	O
,	O
2002	O
)	O
score	O
of	O
the	O
translations	O
.	O
Following	O
the	O
evaluation	O
step	O
in	O
M2M-100	O
,	O
before	O
computing	O
BLEU	O
,	O
we	O
de	O
-	O
tokenize	O
the	O
data	O
and	O
apply	O
standard	O
tokenizers	O
for	O
each	O
language	O
.	O
We	O
use	O
SacreBLEU	O
tokenizer	O
for	O
Chinese	O
,	O
Kytea	O
4	O
for	O
Japanese	O
,	O
and	O
Moses	O
tokenizer	O
5	O
for	O
the	O
rest	O
of	O
the	O
languages	O
.	O
Besides	O
BLEU	O
,	O
we	O
also	O
conduct	O
a	O
human	O
evaluation	O
for	O
a	O
more	O
precise	O
analysis	O
.	O

Hyperparameters	O
.	O
We	O
fine	O
-	O
tune	O
all	O
of	O
our	O
NMT	O
models	O
for	O
4	O
epochs	O
with	O
a	O
batch	O
size	O
of	O
4	O
and	O
a	O
warmup	O
rate	O
of	O
0.2	O
.	O
To	O
avoid	O
over	O
-	O
fitting	O
,	O
we	O
set	O
the	O
early	O
stopping	O
patience	O
on	O
the	O
validation	O
set	O
as	O
2	O
.	O
In	O
the	O
context	O
construction	O
,	O
we	O
randomly	O
select	O
5	O
cell	O
values	O
for	O
each	O
target	O
column	O
.	O
The	O
Adam	O
optimizer	O
(	O
Kingma	O
and	O
Ba	O
,	O
2015	O
)	O
with	O
ß1	O
=	O
0.9	O
,	O
ß2	O
=	O
0.99	O
and	O
✏	O
=	O
1e-8	O
is	O
adopted	O
.	O
We	O
set	O
the	O
number	O
of	O
relation	O
-	O
aware	O
layers	O
as	O
2	O
,	O
and	O
we	O
set	O
the	O
learning	O
rate	O
of	O
the	O
decoder	O
and	O
the	O
relational	O
aware	O
layers	O
as	O
3e-5	O
,	O
and	O
decrease	O
the	O
learning	O
rate	O
of	O
the	O
Transformer	O
encoder	O
to	O
4	O
times	O
and	O
8	O
times	O
smaller	O
for	O
M2M-100	O
and	O
MBart-50M2	O
M	O
respectively	O
.	O

Experimental	O
Results	O

We	O
conduct	O
experiments	O
of	O
translating	O
schema	O
from	O
English	O
(	O
En	O
)	O
to	O
five	O
different	O
languages	O
,	O
including	O
Chinese	O
(	O
Zh	O
)	O
,	O
French	O
(	O
Fr	O
)	O
,	O
German	O
(	O
De	O
)	O
,	O
Spanish	O
(	O
Es	O
)	O
,	O
and	O
Japanese	O
(	O
Ja	O
)	O
.	O
The	O
performances	O
of	O
different	O
translation	O
models	O
are	O
listed	O
in	O
Table	O
4	O
.	O

Overall	O
Performance	O
.	O
The	O
overall	O
performances	O
of	O
two	O
NMT	O
models	O
across	O
five	O
target	O
languages	O
show	O
similar	O
trends	O
.	O
Firstly	O
,	O
compared	O
with	O
Base	O
,	O
which	O
is	O
trained	O
only	O
on	O
plain	O
text	O
,	O
H2H	O
gains	O
significant	O
improvement	O
.	O
For	O
example	O
,	O
H2H	O
based	O
on	O
M2M-100	O
outperforms	O
Base	O
by	O
17.7	O
,	O
24.7	O
,	O
26.7	O
,	O
15.5	O
,	O
and	O
16.6	O
BLEU	O
in	O
translating	O
schema	O
from	O
En	O
to	O
Zh	O
,	O
Es	O
,	O
Fr	O
,	O
De	O
,	O
and	O
Ja	O
,	O
respectively	O
.	O
It	O
demonstrates	O
a	O
big	O
difference	O
between	O
plain	O
text	O
and	O
tabular	O
data	O
,	O
and	O
fine	O
-	O
tuning	O
on	O
schema	O
translation	O
data	O
could	O
alleviate	O
the	O
difference	O
to	O
some	O
extent	O
.	O

Next	O
,	O
we	O
find	O
that	O
,	O
in	O
most	O
situations	O
,	O
the	O
performance	O
of	O
H2H	O
can	O
be	O
further	O
boosted	O
by	O
concatenating	O
the	O
constructed	O
context	O
from	O
the	O
table	O
.	O
Taking	O
H2H+CXT	O
based	O
on	O
M2M-100	O
as	O
an	O
example	O
,	O
comparing	O
with	O
H2H	O
,	O
H2H+CXT	O
obtains	O
2.1	O
,	O
0.6	O
,	O
and	O
1.6	O
points	O
of	O
improvement	O
in	O
En	O
-	O
Zh	O
,	O
En	O
-	O
De	O
,	O
and	O
En	O
-	O
Ja	O
settings	O
,	O
respectively	O
.	O
In	O
terms	O
of	O
H2H+CXT	O
based	O
on	O
MBart-50M2	O
M	O
,	O
the	O
concatenation	O
of	O
context	O
also	O
boosts	O
the	O
BLEU	O
score	O
for	O
translating	O
schema	O
from	O
En	O
to	O
Zh	O
and	O
Es	O
by	O
1.5	O
and	O
1.2	O
.	O
The	O
observations	O
demonstrate	O
the	O
benefits	O
of	O
making	O
good	O
use	O
of	O
the	O
constructed	O
context	O
.	O

However	O
,	O
we	O
also	O
notice	O
that	O
concatenating	O
the	O
context	O
does	O
not	O
help	O
improve	O
the	O
performance	O
of	O
H2H+CXT	O
based	O
on	O
MBart-50M2	O
M	O
and	O
M2M100	O
in	O
the	O
setting	O
of	O
En	O
-	O
De	O
and	O
En	O
-	O
Ja	O
,	O
and	O
the	O
setting	O
of	O
En	O
-	O
Es	O
and	O
En	O
-	O
Fr	O
,	O
respectively	O
.	O
We	O
hypothesize	O
that	O
the	O
decrease	O
of	O
BLEU	O
score	O
comes	O
from	O
the	O
noise	O
brought	O
by	O
the	O
context	O
.	O

There	O
are	O
no	O
significant	O
differences	O
between	O
the	O
performance	O
of	O
H2H+CXT	O
and	O
H2H+CXT+ExtL	O
which	O
has	O
two	O
extra	O
Transformers	O
layers	O
since	O
the	O
pre	O
-	O
trained	O
NMT	O
models	O
have	O
already	O
had	O
12	O
Transformers	O
layers	O
.	O

For	O
example	O
,	O
the	O
H2H+CXT+ExtL	O
model	O
based	O
on	O
M2M100	O
obtains	O
47	O
.	O
1	O
,	O
48.6	O
,	O
53.0	O
,	O
46.6	O
,	O
and	O
40.4	O
BLEU	O
points	O
on	O
En	O
-	O
Zh	O
,	O
En	O
-	O
Es	O
,	O
En	O
-	O
Fr	O
,	O
En	O
-	O
De	O
,	O
and	O
En	O
-	O
Ja	O
,	O
respectively	O
.	O

Finally	O
,	O
equipped	O
with	O
the	O
relation	O
-	O
aware	O
module	O
,	O
CAST	O
can	O
make	O
the	O
best	O
use	O
of	O
the	O
context	O
and	O
obtain	O
significant	O
improvement	O
over	O
H2H	O
across	O
all	O
settings	O
.	O
For	O
models	O
based	O
on	O
M2M-100	O
,	O
CAST	O
outperforms	O
H2H	O
by	O
2.6	O
,	O
1.4	O
,	O
0.3	O
,	O
1.8	O
,	O
and	O
1.9	O
BLEU	O
in	O
En	O
-	O
Zh	O
,	O
En	O
-	O
Es	O
,	O
En	O
-	O
Fr	O
,	O
En	O
-	O
De	O
,	O
and	O
En	O
-	O
Ja	O
,	O
respectively	O
.	O
When	O
it	O
comes	O
to	O
models	O
based	O
on	O
MBart-50M2	O
M	O
,	O
CAST	O
obtains	O
1.6	O
,	O
2.7	O
,	O
1.9	O
,	O
0.9	O
,	O
0.2	O
improvements	O
of	O
BLEU	O
points	O
over	O
H2H	O
in	O
translating	O
schema	O
from	O
En	O
to	O
5	O
target	O
languages	O
.	O
It	O
is	O
also	O
noticeable	O
that	O
CAST	O
can	O
help	O
denoise	O
the	O
concatenated	O
context	O
for	O
H2H+CXT	O
.	O
For	O
instance	O
,	O
CAST	O
based	O
on	O
M2M-100	O
achieves	O
1.5	O
and	O
1.2	O
improvements	O
of	O
BLEU	O
points	O
over	O
H2H+CXT	O
for	O
schema	O
translation	O
from	O
En	O
to	O
Es	O
and	O
Fr	O
respectively	O
.	O
This	O
improvement	O
shows	O
CAST	O
can	O
better	O
model	O
the	O
target	O
header	O
and	O
its	O
context	O
.	O
We	O
also	O
run	O
a	O
Wilcoxon	O
signed	O
-	O
rank	O
tests	O
between	O
CAST	O
and	O
H2H+CXT	O
and	O
the	O
results	O
show	O
the	O
improvement	O
are	O
significant	O
with	O
p	O
<	O
0.05	O
in	O
3	O
out	O
of	O
5	O
languages	O
.	O
For	O
the	O
rest	O
of	O
the	O
languages	O
CAST	O
achieves	O
comparable	O
results	O
.	O

Human	O
Evaluation	O
.	O
Since	O
the	O
machine	O
evaluation	O
metrics	O
can	O
not	O
absolutely	O
make	O
sure	O
whether	O
the	O
predicted	O
result	O
is	O
correct	O
or	O
not	O
,	O
we	O
conduct	O
a	O
human	O
evaluation	O
on	O
the	O
test	O
set	O
for	O
a	O
more	O
precise	O
evaluation	O
.	O
Specifically	O
,	O
we	O
invite	O
two	O
experts	O
to	O
evaluate	O
each	O
language	O
pair	O
.	O
For	O
each	O
case	O
,	O
they	O
compare	O
the	O
machine	O
translation	O
and	O
the	O
human	O
annotation	O
.	O
The	O
label	O
is	O
set	O
as	O
1	O
if	O
they	O
think	O
the	O
translation	O
is	O
equivalent	O
to	O
the	O
annotation	O
,	O
otherwise	O
0	O
.	O
We	O
report	O
the	O
human	O
evaluation	O
results	O
for	O
the	O
Base	O
,	O
H2H	O
,	O
H2H+CXT	O
,	O
and	O
CAST	O
based	O
on	O
M2M-100	O
on	O
the	O
En	O
-	O
Zh	O
setting	O
in	O
Table	O
5	O
.	O
According	O
to	O
human	O
evaluation	O
,	O
H2H	O
achieves	O
14.84	O
%	O
improvement	O
over	O
Base	O
,	O
and	O
the	O
performance	O
is	O
further	O
boosted	O
by	O
3.11	O
%	O
when	O
the	O
context	O
is	O
added	O
.	O
Finally	O
,	O
enhanced	O
by	O
the	O
relationaware	O
structure	O
,	O
CAST	O
obtains	O
2.3	O
%	O
improvement	O
over	O
H2H+CXT	O
,	O
which	O
demonstrates	O
the	O
effectiveness	O
of	O
our	O
approach	O
.	O

Ablation	O
Study	O

We	O
conduct	O
ablation	O
studies	O
on	O
CAST	O
to	O
analyze	O
the	O
contributions	O
of	O
our	O
predefined	O
entity	O
types	O
and	O
structural	O
relationships	O
for	O
context	O
modeling	O
.	O
First	O
,	O
we	O
evaluate	O
the	O
variant	O
of	O
CAST	O
without	O
entity	O
types	O
.	O
Next	O
,	O
we	O
evaluate	O
the	O
performance	O
of	O
CAST	O
,	O
without	O
structural	O
relations	O
.	O
Finally	O
,	O
we	O
erase	O
all	O
kinds	O
of	O
relations	O
in	O
CAST	O
which	O
is	O
identical	O
to	O
H2H+CXT	O
.	O
We	O
report	O
the	O
performance	O
of	O
models	O
based	O
on	O
M2M-100	O
in	O
the	O
setting	O
of	O
En	O
-	O
De	O
and	O
En	O
-	O
Fr	O
in	O
Table	O
6	O
.	O

Firstly	O
,	O
it	O
is	O
clear	O
that	O
erasing	O
entity	O
types	O
decreases	O
the	O
performance	O
of	O
the	O
schema	O
translation	O
Table	O
7	O
:	O
Qualitative	O
analysis	O
for	O
models	O
'	O
performance	O
in	O
schema	O
translation	O
from	O
En	O
to	O
Zh	O
on	O
three	O
kinds	O
of	O
headers	O
.	O
For	O
each	O
predicting	O
result	O
,	O
we	O
add	O
extra	O
explanations	O
for	O
their	O
meanings	O
in	O
the	O
brackets	O
.	O
Results	O
with	O
underline	O
denote	O
the	O
correct	O
translation	O
for	O
the	O
header	O
.	O
models	O
.	O
Comparing	O
CAST	O
(	O
w/o	O
entity	O
type	O
)	O
with	O
CAST	O
,	O
for	O
instance	O
,	O
We	O
can	O
see	O
a	O
0.5	O
and	O
0.5	O
decrease	O
of	O
BLEU	O
for	O
En	O
-	O
De	O
and	O
En	O
-	O
Fr	O
respectively	O
.	O
Secondly	O
,	O
the	O
comparison	O
between	O
CAST	O
(	O
w/o	O
structural	O
relation	O
)	O
and	O
CAST	O
shows	O
that	O
the	O
structure	O
relations	O
also	O
play	O
an	O
important	O
role	O
in	O
bettering	O
the	O
performance	O
of	O
context	O
modeling	O
.	O
As	O
seen	O
in	O
the	O
En	O
-	O
Fr	O
translation	O
setting	O
,	O
CAST(w	O
/	O
o	O
structural	O
relation	O
)	O
obtains	O
a	O
1.0	O
lower	O
BLEU	O
score	O
over	O
CAST	O
.	O
Finally	O
,	O
when	O
erasing	O
both	O
kinds	O
of	O
edges	O
and	O
the	O
models	O
give	O
the	O
lowest	O
performance	O
.	O

Qualitative	O
Analysis	O

In	O
this	O
section	O
,	O
we	O
conduct	O
a	O
qualitative	O
analysis	O
on	O
the	O
effectiveness	O
of	O
CAST	O
based	O
on	O
M2M-100	O
for	O
three	O
types	O
of	O
headers	O
:	O
headers	O
with	O
special	O
tokenization	O
,	O
abbreviation	O
headers	O
,	O
and	O
polysemy	O
headers	O
.	O
We	O
list	O
some	O
of	O
the	O
example	O
translations	O
in	O
Table	O
7	O
.	O

By	O
comparing	O
the	O
translations	O
for	O
headers	O
with	O
special	O
tokenization	O
,	O
we	O
can	O
see	O
that	O
all	O
fine	O
-	O
tuned	O
models	O
,	O
including	O
H2H	O
,	O
H2H+CXT	O
,	O
and	O
CAST	O
can	O
accurately	O
translate	O
headers	O
in	O
CamelCase	O
or	O
underscore	O
tokenizations	O
,	O
while	O
Base	O
fails	O
to	O
skip	O
the	O
underscore	O
and	O
can	O
not	O
translate	O
"	O
Debt	O
"	O
in	O
the	O
middle	O
of	O
"	O
AccessedDebtService	O
"	O
.	O

For	O
the	O
abbreviation	O
headers	O
,	O
when	O
translating	O
"	O
OS	O
"	O
(	O
the	O
abbreviation	O
of	O
operation	O
system	O
)	O
and	O
"	O
Jan	O
"	O
(	O
the	O
abbreviation	O
of	O
January	O
)	O
,	O
both	O
Base	O
and	O
H2H	O
fail	O
to	O
get	O
the	O
correct	O
result	O
.	O
However	O
,	O
being	O
aware	O
of	O
the	O
context	O
of	O
"	O
Jan	O
"	O
(	O
e.g.	O
,	O
Feb	O
,	O
Mar	O
and	O
Apr	O
,	O
etc	O
.	O
)	O
and	O
"	O
OS	O
"	O
(	O
e.g.	O
,	O
Computer	O
,	O
System	O
,	O
and	O
Core	O
,	O
etc	O
.	O
)	O
,	O
H2H+CXT	O
and	O
CAST	O
can	O
better	O
understand	O
and	O
translate	O
the	O
abbreviations	O
.	O

When	O
it	O
comes	O
to	O
the	O
polysemy	O
headers	O
,	O
with	O
the	O
help	O
of	O
context	O
like	O
"	O
Height	O
"	O
,	O
"	O
Width	O
"	O
and	O
"	O
Depth	O
"	O
,	O
H2H+CXT	O
and	O
CAST	O
can	O
disambiguate	O
polysemy	O
header	O
"	O
Area	O
"	O
from	O
region	O
or	O
zone	O
to	O
acreage	O
.	O
For	O
header	O
"	O
Volume	O
"	O
,	O
However	O
,	O
H2H+CXT	O
copies	O
the	O
source	O
language	O
column	O
,	O
which	O
is	O
not	O
a	O
valid	O
translation	O
,	O
because	O
the	O
translator	O
is	O
disturbed	O
by	O
the	O
context	O
.	O
On	O
the	O
other	O
hand	O
,	O
with	O
the	O
help	O
of	O
the	O
relational	O
-	O
aware	O
transformer	O
encoder	O
,	O
CAST	O
generates	O
a	O
proper	O
translation	O
for	O
"	O
Volume	O
"	O
as	O
the	O
capacity	O
of	O
the	O
engine	O
.	O
Affected	O
by	O
the	O
context	O
,	O
H2H+CXT	O
only	O
translates	O
part	O
of	O
the	O
information	O
from	O
header	O
'	O
Film.1	O
'	O
and	O
'	O
Rank	O
of	O
the	O
year	O
'	O
,	O
while	O
M2M-100	O
,	O
H2H	O
,	O
and	O
CAST	O
give	O
an	O
appropriate	O
translation	O
.	O

Related	O
Work	O

With	O
the	O
developments	O
of	O
Neural	O
Machine	O
Translation	O
(	O
NMT	O
)	O
systems	O
(	O
Sutskever	O
et	O
al	O
.	O
,	O
2014;Bahdanau	O
et	O
al	O
.	O
,	O
2015	O
)	O
,	O
tremendous	O
success	O
has	O
been	O
achieved	O
by	O
existing	O
studies	O
on	O
machine	O
translation	O
tasks	O
.	O
For	O
instance	O
,	O
Vaswani	O
et	O
al	O
.	O
(	O
2017	O
)	O
greatly	O
improved	O
bilingual	O
machine	O
translation	O
systems	O
with	O
the	O
Transformer	O
architectures	O
,	O
(	O
Edunov	O
et	O
al	O
.	O
,	O
2018	O
)	O
achieved	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
on	O
the	O
WMT	O
'	O
14	O
English	O
-	O
German	O
tasks	O
with	O
back	O
-	O
translations	O
augmentation	O
,	O
Weng	O
et	O
al	O
.	O
(	O
2020	O
)	O
and	O
Yang	O
et	O
al	O
.	O
(	O
2020	O
)	O
explored	O
ways	O
to	O
boost	O
the	O
performance	O
of	O
NMT	O
systems	O
with	O
pre	O
-	O
trained	O
language	O
models	O
.	O
Recent	O
works	O
saw	O
the	O
potential	O
to	O
improve	O
NMT	O
models	O
in	O
many	O
-	O
to	O
-	O
many	O
settings	O
and	O
proposed	O
models	O
that	O
can	O
perform	O
machine	O
translation	O
on	O
various	O
language	O
pairs	O
.	O
While	O
the	O
above	O
-	O
mentioned	O
studies	O
focus	O
on	O
sentence	O
-	O
level	O
translation	O
in	O
plain	O
text	O
,	O
they	O
are	O
not	O
suitable	O
for	O
schema	O
translation	O
.	O

A	O
line	O
of	O
machine	O
translation	O
research	O
closely	O
related	O
to	O
our	O
task	O
is	O
the	O
phrase	O
-	O
to	O
-	O
phrase	O
translation	O
,	O
which	O
considers	O
phrases	O
in	O
multi	O
-	O
word	O
expressions	O
as	O
their	O
translation	O
unit	O
.	O
Traditional	O
phrase	O
-	O
based	O
SMT	O
models	O
(	O
Koehn	O
et	O
al	O
.	O
,	O
2007;Haddow	O
et	O
al	O
.	O
,	O
2015	O
)	O
get	O
phrase	O
table	O
translation	O
probabilities	O
by	O
counting	O
phrase	O
occurrences	O
and	O
use	O
local	O
context	O
through	O
a	O
smoothed	O
n	O
-	O
gram	O
language	O
model	O
.	O
Recently	O
,	O
some	O
works	O
explore	O
ways	O
to	O
adapt	O
NMT	O
models	O
for	O
phrase	O
translation	O
.	O
For	O
example	O
,	O
Wang	O
et	O
al	O
.	O
(	O
2017	O
)	O
combined	O
the	O
phrase	O
-	O
based	O
statistical	O
machine	O
translation	O
(	O
SMT	O
)	O
model	O
into	O
NMT	O
and	O
shown	O
significant	O
improvements	O
on	O
Chineseto	O
-	O
English	O
translation	O
data	O
,	O
explored	O
the	O
use	O
of	O
phrase	O
structures	O
for	O
NMT	O
systems	O
by	O
modeling	O
phrases	O
in	O
target	O
language	O
sequences	O
,	O
and	O
Feng	O
et	O
al	O
.	O
(	O
2018	O
)	O
used	O
a	O
phrase	O
attention	O
mechanism	O
to	O
enhance	O
the	O
decoder	O
in	O
relevant	O
source	O
segment	O
recognition	O
.	O
The	O
main	O
differences	O
between	O
these	O
studies	O
and	O
our	O
work	O
are	O
:	O
(	O
1	O
)	O
we	O
do	O
not	O
rely	O
on	O
external	O
phrase	O
dictionaries	O
or	O
phrase	O
tables	O
;	O
and	O
(	O
2	O
)	O
we	O
study	O
how	O
to	O
make	O
use	O
of	O
the	O
schema	O
context	O
for	O
word	O
-	O
sense	O
disambiguation	O
in	O
the	O
schema	O
translation	O
scenario	O
.	O

Context	O
-	O
aware	O
schema	O
encoding	O
has	O
received	O
considerable	O
attention	O
in	O
both	O
recent	O
semantic	O
parsing	O
literature	O
(	O
Hwang	O
et	O
al	O
.	O
,	O
2019;Gong	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
Table	O
-	O
to	O
-	O
Text	O
literature	O
(	O
Gong	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
In	O
general	O
,	O
there	O
are	O
two	O
sorts	O
of	O
techniques	O
:	O
1	O
)	O
.	O
add	O
additional	O
entity	O
type	O
embedding	O
and	O
special	O
separator	O
token	O
from	O
the	O
input	O
sequence	O
to	O
distinguish	O
the	O
table	O
structure	O
(	O
i.e.	O
,	O
Type	O
-	O
SQL	O
and	O
IRNET	O
)	O
;	O
2	O
)	O
.	O
encode	O
the	O
schema	O
as	O
a	O
directed	O
graph	O
.	O
For	O
example	O
,	O
Bogin	O
et	O
al	O
.	O
(	O
2019	O
)	O
use	O
a	O
Graph	O
Neural	O
Network	O
(	O
Scarselli	O
et	O
al	O
.	O
,	O
2008	O
)	O
,	O
and	O
;	O
Shaw	O
et	O
al	O
.	O
(	O
2019	O
)	O
use	O
a	O
transformer	O
self	O
-	O
attention	O
mechanism	O
to	O
encode	O
the	O
schema	O
over	O
predefined	O
schema	O
relationships	O
.	O
Unlike	O
these	O
works	O
,	O
we	O
explore	O
the	O
suitability	O
of	O
schema	O
encoding	O
techniques	O
for	O
the	O
newly	O
proposed	O
schema	O
translation	O
task	O
.	O

Conclusion	O

In	O
this	O
paper	O
,	O
we	O
propose	O
a	O
new	O
challenging	O
translation	O
task	O
called	O
schema	O
translation	O
,	O
and	O
construct	O
the	O
first	O
parallel	O
dataset	O
for	O
this	O
task	O
.	O
To	O
address	O
the	O
challenges	O
for	O
this	O
new	O
task	O
,	O
we	O
propose	O
CAST	O
,	O
which	O
uses	O
a	O
relational	O
-	O
aware	O
transformer	O
to	O
encode	O
a	O
header	O
and	O
its	O
context	O
over	O
predefined	O
relationships	O
,	O
making	O
it	O
aware	O
of	O
the	O
table	O
context	O
.	O

Ethical	O
Considerations	O

The	O
schema	O
translation	O
dataset	O
presented	O
in	O
this	O
work	O
is	O
a	O
free	O
and	O
open	O
resource	O
for	O
the	O
community	O
to	O
study	O
the	O
newly	O
proposed	O
translation	O
task	O
.	O
English	O
tables	O
collected	O
are	O
from	O
three	O
sources	O
.	O
First	O
,	O
we	O
collect	O
all	O
tables	O
from	O
the	O
WikiTableQuestions	O
dataset	O
(	O
Pasupat	O
and	O
Liang	O
,	O
2015	O
)	O
,	O
which	O
is	O
a	O
free	O
and	O
open	O
dataset	O
for	O
the	O
research	O
of	O
question	O
answering	O
task	O
on	O
semi	O
-	O
structured	O
HTML	O
ta	O
-	O
bles	O
.	O
Since	O
all	O
of	O
the	O
tables	O
are	O
collected	O
from	O
open	O
-	O
access	O
Wikipedia	O
pages	O
,	O
there	O
is	O
no	O
privacy	O
issue	O
.	O
Second	O
,	O
we	O
collect	O
176	O
English	O
tables	O
from	O
the	O
search	O
engines	O
which	O
are	O
also	O
publicly	O
available	O
and	O
do	O
not	O
contain	O
personal	O
data	O
.	O
To	O
Further	O
enlarge	O
our	O
dataset	O
,	O
we	O
select	O
all	O
tables	O
from	O
the	O
training	O
set	O
and	O
development	O
set	O
of	O
the	O
Spider	O
dataset	O
(	O
Yu	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
which	O
is	O
also	O
a	O
free	O
and	O
open	O
dataset	O
for	O
research	O
use	O
.	O
Since	O
the	O
tables	O
from	O
the	O
Spider	O
dataset	O
are	O
mainly	O
collected	O
from	O
openaccess	O
online	O
csv	O
files	O
,	O
college	O
database	O
courses	O
and	O
SQL	O
websites	O
,	O
there	O
is	O
no	O
privacy	O
issue	O
either	O
.	O
For	O
the	O
translation	O
step	O
,	O
we	O
hire	O
professional	O
translators	O
to	O
translate	O
the	O
collected	O
English	O
tables	O
to	O
five	O
target	O
languages	O
and	O
the	O
details	O
can	O
be	O
found	O
in	O
Section	O
2	O
.	O

All	O
the	O
experiments	O
with	O
NMT	O
models	O
in	O
this	O
paper	O
can	O
be	O
run	O
on	O
a	O
single	O
Tesla	O
V100	O
GPU	O
.	O
On	O
average	O
,	O
the	O
training	O
process	O
of	O
models	O
in	O
different	O
languages	O
can	O
be	O
finished	O
in	O
four	O
hours	O
.	O
We	O
implement	O
our	O
model	O
with	O
the	O
Transformer	O
6	O
tools	O
in	O
Pytorch	O
7	O
,	O
and	O
the	O
data	O
will	O
be	O
released	O
with	O
the	O
paper	O
.	O

Multimodal	O
Quality	O
Estimation	O
for	O
Machine	O
Translation	O

We	O
propose	O
approaches	O
to	O
Quality	O
Estimation	O
(	O
QE	O
)	O
for	O
Machine	O
Translation	O
that	O
explore	O
both	O
text	O
and	O
visual	O
modalities	O
for	O
Multimodal	O
QE	O
.	O
We	O
compare	O
various	O
multimodality	O
integration	O
and	O
fusion	O
strategies	O
.	O
For	O
both	O
sentence	O
-	O
level	O
and	O
document	O
-	O
level	O
predictions	O
,	O
we	O
show	O
that	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
neural	O
and	O
feature	O
-	O
based	O
QE	O
frameworks	O
obtain	O
better	O
results	O
when	O
using	O
the	O
additional	O
modality	O
.	O

Quality	O
Estimation	O
(	O
QE	O
)	O
for	O
Machine	O
Translation	O
(	O
MT	O
)	O
(	O
Blatz	O
et	O
al	O
.	O
,	O
2004;Specia	O
et	O
al	O
.	O
,	O
2009	O
)	O
aims	O
to	O
predict	O
the	O
quality	O
of	O
a	O
machine	O
-	O
translated	O
text	O
without	O
using	O
reference	O
translations	O
.	O
It	O
estimates	O
a	O
label	O
(	O
a	O
category	O
,	O
such	O
as	O
'	O
good	O
'	O
or	O
'	O
bad	O
'	O
,	O
or	O
a	O
numerical	O
score	O
)	O
for	O
a	O
translation	O
,	O
given	O
text	O
in	O
a	O
source	O
language	O
and	O
its	O
machine	O
translation	O
in	O
a	O
target	O
language	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2018b	O
)	O
.	O
QE	O
can	O
operate	O
at	O
different	O
linguistic	O
levels	O
,	O
including	O
sentence	O
and	O
document	O
levels	O
.	O
Sentence	O
-	O
level	O
QE	O
estimates	O
the	O
translation	O
quality	O
of	O
a	O
whole	O
sentence	O
,	O
while	O
document	O
-	O
level	O
QE	O
predicts	O
the	O
translation	O
quality	O
of	O
an	O
entire	O
document	O
,	O
even	O
though	O
in	O
practice	O
in	O
literature	O
the	O
documents	O
have	O
been	O
limited	O
to	O
a	O
small	O
set	O
of	O
3	O
-	O
5	O
sentences	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2018b	O
)	O
.	O

Table	O
1	O
:	O
Example	O
of	O
incorrectly	O
machine	O
-	O
translated	O
text	O
:	O
the	O
word	O
shorts	O
is	O
used	O
to	O
indicate	O
short	O
trousers	O
,	O
but	O
gets	O
translated	O
in	O
French	O
as	O
court	O
,	O
the	O
adjective	O
short	O
.	O
Here	O
multimodality	O
could	O
help	O
to	O
detect	O
the	O
error	O
(	O
extracted	O
from	O
the	O
Amazon	O
Reviews	O
Dataset	O
of	O
McAuley	O
et	O
al	O
.	O
,	O
2015	O
)	O
.	O
creasingly	O
accompanied	O
with	O
visual	O
elements	O
such	O
as	O
images	O
or	O
videos	O
,	O
especially	O
in	O
social	O
media	O
but	O
also	O
in	O
domains	O
such	O
as	O
e	O
-	O
commerce	O
.	O
Multimodality	O
has	O
not	O
yet	O
been	O
applied	O
to	O
QE	O
.	O
Table	O
1	O
shows	O
an	O
example	O
from	O
our	O
e	O
-	O
commerce	O
dataset	O
in	O
which	O
multimodality	O
could	O
help	O
to	O
improve	O
QE	O
.	O
Here	O
,	O
the	O
English	O
noun	O
shorts	O
is	O
translated	O
by	O
the	O
adjective	O
court	O
(	O
for	O
the	O
adjective	O
short	O
)	O
in	O
French	O
,	O
which	O
is	O
a	O
possible	O
translation	O
out	O
of	O
context	O
.	O
However	O
,	O
as	O
the	O
corresponding	O
product	O
image	O
shows	O
,	O
this	O
product	O
is	O
an	O
item	O
of	O
clothing	O
,	O
and	O
thus	O
the	O
machine	O
translation	O
is	O
incorrect	O
.	O
External	O
information	O
can	O
hence	O
help	O
identify	O
mismatches	O
between	O
translations	O
which	O
are	O
difficult	O
to	O
find	O
within	O
the	O
text	O
.	O
Progress	O
in	O
QE	O
is	O
mostly	O
benchmarked	O
as	O
part	O
of	O
the	O
Conference	O
on	O
Machine	O
Translation	O
(	O
WMT	O
)	O
Shared	O
Task	O
on	O
QE	O
.	O
This	O
paper	O
is	O
based	O
on	O
data	O
from	O
the	O
WMT'18	O
edition	O
's	O
Task	O
4	O
-documentlevel	O
QE	O
.	O
This	O
Task	O
4	O
aims	O
to	O
predict	O
a	O
translation	O
quality	O
score	O
for	O
short	O
documents	O
based	O
on	O
the	O
number	O
and	O
the	O
severity	O
of	O
translation	O
errors	O
at	O
the	O
word	O
level	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2018a	O
)	O
.	O
This	O
data	O
was	O
chosen	O
as	O
it	O
is	O
the	O
only	O
one	O
for	O
which	O
meta	O
information	O
(	O
images	O
in	O
this	O
case	O
)	O
is	O
available	O
.	O
We	O
extend	O
this	O
dataset	O
by	O
computing	O
scores	O
for	O
each	O
sentence	O
for	O
a	O
sentence	O
-	O
level	O
prediction	O
task	O
.	O
We	O
consider	O
both	O
feature	O
-	O
based	O
and	O
neural	O
state	O
-	O
of	O
-	O
theart	O
models	O
for	O
QE	O
.	O
Having	O
these	O
as	O
our	O
starting	O
points	O
,	O
we	O
propose	O
different	O
ways	O
to	O
integrate	O
the	O
visual	O
modality	O
.	O

The	O
main	O
contributions	O
of	O
this	O
paper	O
are	O
as	O
follows	O
:	O
(	O
i	O
)	O
we	O
introduce	O
the	O
task	O
of	O
Multimodal	O
QE	O
(	O
MQE	O
)	O
for	O
MT	O
as	O
an	O
attempt	O
to	O
improve	O
QE	O
by	O
using	O
external	O
sources	O
of	O
information	O
,	O
namely	O
images	O
;	O
(	O
ii	O
)	O
we	O
propose	O
several	O
ways	O
of	O
incorporating	O
visual	O
information	O
in	O
neural	O
-	O
based	O
and	O
featurebased	O
QE	O
architectures	O
;	O
and	O
(	O
iii	O
)	O
we	O
achieve	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
performance	O
for	O
such	O
architectures	O
in	O
document	O
and	O
sentence	O
-	O
level	O
QE	O
.	O

QE	O
Frameworks	O
and	O
Models	O

We	O
explore	O
feature	O
-	O
based	O
and	O
neural	O
-	O
based	O
models	O
from	O
two	O
open	O
-	O
source	O
frameworks	O
:	O
QuEst++	O
:	O
QuEst++	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2015	O
)	O
is	O
a	O
feature	O
-	O
based	O
QE	O
framework	O
composed	O
of	O
two	O
modules	O
:	O
a	O
feature	O
extractor	O
module	O
,	O
to	O
extract	O
the	O
relevant	O
QE	O
features	O
from	O
both	O
the	O
source	O
sentences	O
and	O
their	O
translations	O
,	O
and	O
a	O
machine	O
learning	O
module	O
.	O
We	O
only	O
use	O
this	O
framework	O
for	O
our	O
experiments	O
on	O
document	O
-	O
level	O
QE	O
,	O
since	O
it	O
does	O
not	O
perform	O
well	O
enough	O
for	O
sentence	O
-	O
level	O
prediction	O
.	O
We	O
use	O
the	O
same	O
model	O
(	O
Support	O
Vector	O
Regression	O
)	O
,	O
hyperparameters	O
and	O
feature	O
settings	O
as	O
the	O
baseline	O
model	O
for	O
the	O
document	O
-	O
level	O
QE	O
task	O
at	O
WMT'18	O
.	O

deepQuest	O
:	O
deepQuest	O
(	O
I	O
ve	O
et	O
al	O
.	O
,	O
2018	O
)	O
is	O
a	O
neural	O
-	O
based	O
framework	O
that	O
provides	O
state	O
-	O
of	O
-	O
theart	O
models	O
for	O
multi	O
-	O
level	O
QE	O
.	O
We	O
use	O
the	O
BiRNN	O
model	O
,	O
a	O
light	O
-	O
weight	O
architecture	O
which	O
can	O
be	O
trained	O
at	O
either	O
sentence	O
or	O
document	O
level	O
.	O

The	O
BiRNN	O
model	O
uses	O
an	O
encoder	O
-	O
decoder	O
architecture	O
:	O
it	O
takes	O
on	O
its	O
input	O
both	O
the	O
source	O
sentence	O
and	O
its	O
translation	O
which	O
are	O
encoded	O
separately	O
by	O
two	O
independent	O
bi	O
-	O
directional	O
Recurrent	O
Neural	O
Networks	O
(	O
RNNs	O
)	O
.	O
The	O
two	O
resulting	O
sentence	O
representations	O
are	O
then	O
concatenated	O
as	O
a	O
weighted	O
sum	O
of	O
their	O
word	O
vectors	O
,	O
generated	O
by	O
an	O
attention	O
mechanism	O
.	O
For	O
sentence	O
-	O
level	O
predictions	O
,	O
the	O
weighted	O
representation	O
of	O
the	O
two	O
input	O
sentences	O
is	O
passed	O
through	O
a	O
dense	O
layer	O
with	O
sigmoid	O
activation	O
to	O
generate	O
the	O
quality	O
estimates	O
.	O
For	O
document	O
-	O
level	O
predictions	O
,	O
the	O
final	O
representation	O
of	O
a	O
document	O
is	O
generated	O
by	O
a	O
second	O
attention	O
mechanism	O
,	O
as	O
the	O
weighted	O
sum	O
of	O
the	O
weighted	O
sentence	O
-	O
level	O
representations	O
of	O
all	O
the	O
sentences	O
within	O
the	O
document	O
.	O
The	O
resulting	O
document	O
-	O
level	O
representation	O
is	O
then	O
passed	O
through	O
a	O
dense	O
layer	O
with	O
sigmoid	O
activation	O
to	O
generate	O
the	O
quality	O
estimates	O
.	O

Additionally	O
,	O
we	O
propose	O
and	O
experiment	O
with	O
BERT	O
-	O
BiRNN	O
,	O
a	O
variant	O
of	O
the	O
BiRNN	O
model	O
.	O
Rather	O
than	O
training	O
the	O
token	O
embeddings	O
with	O
the	O
task	O
at	O
hand	O
,	O
we	O
use	O
large	O
-	O
scale	O
pre	O
-	O
trained	O
token	O
-	O
level	O
representations	O
from	O
the	O
multilingual	O
cased	O
base	O
BERT	O
model	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
During	O
training	O
,	O
the	O
BERT	O
model	O
is	O
fine	O
-	O
tuned	O
by	O
unfreezing	O
the	O
weights	O
of	O
the	O
last	O
four	O
hidden	O
layers	O
along	O
with	O
the	O
token	O
embedding	O
layer	O
.	O
This	O
performs	O
comparably	O
to	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
predictorestimator	O
neural	O
model	O
in	O
Kepler	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O

WMT'18	O
QE	O
Task	O
4	O
data	O
:	O
This	O
dataset	O
was	O
created	O
for	O
the	O
document	O
-	O
level	O
track	O
.	O
It	O
contains	O
a	O
sample	O
of	O
products	O
from	O
the	O
Amazon	O
Reviews	O
Dataset	O
(	O
McAuley	O
et	O
al	O
.	O
,	O
2015	O
)	O
taken	O
from	O
the	O
Sports	O
&	O
Outdoors	O
category	O
.	O
'	O
Documents	O
'	O
consist	O
of	O
the	O
English	O
product	O
title	O
and	O
its	O
description	O
,	O
its	O
French	O
machinetranslation	O
and	O
a	O
numerical	O
score	O
to	O
predict	O
,	O
namely	O
the	O
MQM	O
score	O
(	O
Multidimensional	O
Quality	O
Metrics	O
)	O
(	O
Lommel	O
et	O
al	O
.	O
,	O
2014	O
)	O
.	O
This	O
score	O
is	O
computed	O
by	O
annotating	O
and	O
weighting	O
each	O
word	O
-	O
level	O
translation	O
error	O
according	O
to	O
its	O
severity	O
(	O
minor	O
,	O
major	O
and	O
critical	O
):	O

MQM	O
Score	O
=	O
1	O
−	O
n	O
min	O
+	O
5n	O
maj	O
+	O
10n	O
cri	O
n	O

For	O
the	O
sentence	O
-	O
level	O
QE	O
task	O
,	O
each	O
document	O
of	O
the	O
dataset	O
was	O
split	O
into	O
sentences	O
(	O
lines	O
)	O
,	O
where	O
every	O
sentence	O
has	O
its	O
corresponding	O
MQM	O
score	O
computed	O
in	O
the	O
same	O
way	O
as	O
for	O
the	O
document	O
.	O
We	O
note	O
that	O
this	O
variant	O
is	O
different	O
from	O
the	O
official	O
sentence	O
-	O
level	O
track	O
at	O
WMT	O
since	O
for	O
that	O
task	O
visual	O
information	O
is	O
not	O
available	O
.	O

Text	O
features	O
:	O
For	O
the	O
feature	O
-	O
based	O
approach	O
,	O
we	O
extract	O
the	O
same	O
15	O
features	O
as	O
those	O
for	O
the	O
baseline	O
of	O
WMT'18	O
at	O
document	O
level	O
.	O
For	O
the	O
neural	O
-	O
based	O
approaches	O
,	O
text	O
features	O
are	O
either	O
the	O
learned	O
word	O
embeddings	O
(	O
BiRNN	O
)	O
or	O
pre	O
-	O
trained	O
word	O
embeddings	O
(	O
BERT	O
-	O
BiRNN	O
)	O
.	O

Multimodal	O
QE	O

We	O
propose	O
different	O
ways	O
to	O
integrate	O
visual	O
features	O
in	O
our	O
two	O
monomodal	O
QE	O
approaches	O
(	O
Sections	O
3.1	O
and	O
3.2	O
)	O
.	O
We	O
compare	O
each	O
proposed	O
model	O
with	O
its	O
monomodal	O
QE	O
counterpart	O
as	O
baseline	O
,	O
both	O
using	O
the	O
same	O
hyperparameters	O
.	O

Multimodal	O
feature	O
-	O
based	O
QE	O

The	O
feature	O
-	O
based	O
textual	O
features	O
contain	O
15	O
numerical	O
scores	O
,	O
while	O
the	O
visual	O
feature	O
vector	O
contains	O
4,096	O
dimensions	O
.	O
To	O
avoid	O
over	O
-	O
weighting	O
the	O
visual	O
features	O
,	O
we	O
reduce	O
their	O
dimensionality	O
using	O
Principal	O
Component	O
Analysis	O
(	O
PCA	O
)	O
.	O
We	O
consider	O
up	O
to	O
15	O
principal	O
components	O
in	O
order	O
to	O
keep	O
a	O
balance	O
between	O
the	O
visual	O
features	O
and	O
the	O
15	O
text	O
features	O
from	O
QuEst++	O
.	O
We	O
choose	O
the	O
final	O
number	O
of	O
principal	O
components	O
to	O
keep	O
according	O
to	O
the	O
explained	O
variance	O
with	O
the	O
PCA	O
,	O
so	O
this	O
number	O
is	O
treated	O
as	O
a	O
hyperparameter	O
.	O
After	O
analysing	O
the	O
explained	O
variance	O
for	O
up	O
to	O
15	O
kept	O
principal	O
components	O
(	O
see	O
Figure	O
4	O
in	O
Appendix	O
)	O
,	O
we	O
selected	O
six	O
numbers	O
of	O
principal	O
components	O
to	O
train	O
QE	O
models	O
with	O
(	O
1	O
,	O
2	O
,	O
3	O
,	O
5	O
,	O
10	O
,	O
and	O
15	O
)	O
.	O
As	O
fusion	O
strategy	O
,	O
we	O
concatenate	O
the	O
two	O
feature	O
vectors	O
.	O

Multimodal	O
neural	O
-	O
based	O
QE	O

Multimodality	O
is	O
achieved	O
with	O
two	O
changes	O
in	O
our	O
monomodal	O
models	O
:	O
multimodality	O
integration	O
(	O
where	O
to	O
integrate	O
the	O
visual	O
features	O
in	O
the	O
architecture	O
)	O
,	O
and	O
fusion	O
strategy	O
(	O
how	O
to	O
fuse	O
the	O
visual	O
and	O
textual	O
features	O
)	O
.	O
We	O
propose	O
the	O
following	O
places	O
to	O
integrate	O
the	O
visual	O
feature	O
vector	O
into	O
the	O
BiRNN	O
architecture	O
:	O

•	O
annot	O
-the	O
visual	O
feature	O
vector	O
is	O
used	O
after	O
the	O
encoding	O
of	O
the	O
two	O
input	O
sentences	O
by	O
the	O
two	O
bi	O
-	O
directional	O
RNNs	O
;	O
•	O
last	O
-the	O
visual	O
feature	O
vector	O
is	O
used	O
just	O
before	O
the	O
last	O
layer	O
.	O

Figure	O
1	O
presents	O
the	O
high	O
-	O
level	O
architecture	O
of	O
the	O
document	O
-	O
level	O
BiRNN	O
model	O
,	O
with	O
the	O
various	O
multimodality	O
integration	O
and	O
fusion	O
approaches	O
.	O

We	O
use	O
the	O
standard	O
training	O
,	O
development	O
and	O
test	O
datasets	O
from	O
the	O
WMT'18	O
Task	O
4	O
track	O
.	O
For	O
feature	O
-	O
based	O
systems	O
,	O
we	O
follow	O
the	O
built	O
-	O
in	O
crossvalidation	O
in	O
QuEst++	O
,	O
and	O
train	O
a	O
single	O
model	O
with	O
the	O
hyperparameters	O
found	O
by	O
cross	O
-	O
validation	O
.	O
For	O
neural	O
-	O
based	O
models	O
,	O
we	O
use	O
early	O
-	O
stopping	O
with	O
a	O
patience	O
of	O
10	O
to	O
avoid	O
over	O
-	O
fitting	O
,	O
and	O
all	O
reported	O
figures	O
are	O
averaged	O
over	O
5	O
runs	O
corresponding	O
to	O
different	O
seeds	O
.	O

We	O
follow	O
the	O
evaluation	O
method	O
of	O
the	O
WMT	O
QE	O
tasks	O
:	O
Pearson	O
's	O
r	O
correlation	O
as	O
the	O
main	O
metric	O
(	O
Graham	O
,	O
2015	O
)	O
,	O
Mean	O
-	O
Absolute	O
Error	O
(	O
MAE	O
)	O
and	O
Root	O
-	O
Mean	O
-	O
Squared	O
Error	O
(	O
RMSE	O
)	O
as	O
secondary	O
metrics	O
.	O
For	O
statistical	O
significance	O
on	O
Pearson	O
's	O
r	O
,	O
we	O
compute	O
Williams	O
test	O
(	O
Williams	O
,	O
1959	O
)	O
as	O
suggested	O
by	O
Graham	O
and	O
Baldwin	O
(	O
2014	O
)	O
.	O

For	O
all	O
neural	O
-	O
based	O
models	O
,	O
we	O
experiment	O
with	O
the	O
all	O
three	O
integration	O
strategies	O
(	O
'	O
embed	O
'	O
,	O
'	O
annot	O
'	O
and	O
'	O
last	O
'	O
)	O
and	O
all	O
three	O
fusion	O
strategies	O
(	O
'	O
conc	O
'	O
,	O
'	O
mult	O
'	O
and	O
'	O
mult2	O
'	O
)	O
presented	O
in	O
Section	O
3.2	O
.	O
This	O
leads	O
to	O
6	O
multimodal	O
models	O
for	O
each	O
BiRNN	O
and	O
BERT	O
-	O
BiRNN	O
.	O
In	O
Tables	O
2	O
and	O
4	O
,	O
as	O
well	O
as	O
in	O
Figures	O
2	O
and	O
3	O
,	O
we	O
report	O
the	O
top	O
three	O
performing	O
models	O
.	O
We	O
refer	O
the	O
reader	O
to	O
the	O
Appendix	O
for	O
the	O
full	O
set	O
of	O
results	O
.	O

Sentence	O
-	O
level	O
MQE	O

The	O
first	O
part	O
of	O
Table	O
2	O
presents	O
the	O
results	O
for	O
sentence	O
-	O
level	O
multimodal	O
QE	O
with	O
BiRNN	O
.	O
The	O
best	O
model	O
is	O
BiRNN+Vis	O
-	O
embed	O
-	O
mult2	O
,	O
achieving	O
a	O
Pearson	O
's	O
r	O
of	O
0.535	O
,	O
significantly	O
outperforming	O
the	O
baseline	O
(	O
p	O
-	O
value<0.01	O
)	O
.	O
Visual	O
features	O
can	O
,	O
therefore	O
,	O
help	O
to	O
improve	O
the	O
performance	O
of	O
sentence	O
-	O
level	O
neural	O
-	O
based	O
QE	O
systems	O
significantly	O
.	O

Figure	O
2	O
presents	O
the	O
result	O
of	O
Williams	O
significance	O
test	O
for	O
BiRNN	O
model	O
variants	O
.	O
It	O
is	O
a	O
correlation	O
matrix	O
that	O
can	O
be	O
read	O
as	O
follows	O
:	O
the	O
value	O
in	O
cell	O
(	O
i	O
,	O
j	O
)	O
is	O
the	O
p	O
-	O
value	O
of	O
Williams	O
test	O
for	O
the	O
change	O
in	O
performance	O
of	O
the	O
model	O
at	O
row	O
i	O
compared	O
to	O
the	O
model	O
at	O
column	O
j	O
(	O
Graham	O
,	O
2015	O
)	O
.	O

With	O
the	O
pre	O
-	O
trained	O
token	O
-	O
level	O
representations	O
from	O
BERT	O
(	O
second	O
half	O
of	O
Table	O
2	O
)	O
,	O
the	O
best	O
model	O
is	O
BERT	O
-	O
BiRNN+Vis	O
-	O
annot	O
-	O
mult	O
,	O
achieving	O
a	O
Pear-	O
BERT	O
-	O
BiRNN	O
)	O
and	O
their	O
respective	O
top-3	O
best	O
performing	O
multimodal	O
variants	O
(	O
+	O
Vis	O
)	O
.	O
We	O
refer	O
the	O
reader	O
to	O
the	O
Appendix	O
for	O
the	O
full	O
set	O
of	O
results	O
.	O
Here	O
,	O
BERT	O
,	O
ann	O
-	O
mul	O
and	O
emb	O
-	O
mul2	O
correspond	O
to	O
the	O
BERT	O
-	O
BiRNN	O
,	O
the	O
BERT	O
-	O
BiRNN+Vis	O
-	O
annot	O
-	O
mult	O
and	O
the	O
BiRNN+Vis	O
-	O
embed	O
-	O
mult2	O
models	O
of	O
Table	O
2	O
.	O

son	O
's	O
r	O
of	O
0.602	O
.	O
This	O
shows	O
that	O
even	O
when	O
using	O
better	O
word	O
presentations	O
,	O
the	O
visual	O
features	O
help	O
to	O
get	O
further	O
(	O
albeit	O
modest	O
)	O
improvements	O
.	O
Table	O
3	O
shows	O
an	O
example	O
of	O
predicted	O
scores	O
at	O
the	O
sentence	O
-	O
level	O
for	O
the	O
baseline	O
model	O
(	O
BiRNN	O
)	O
and	O
for	O
the	O
best	O
multimodal	O
BiRNN	O
model	O
(	O
BiRNN+Vis	O
-	O
embed	O
-	O
mult2	O
)	O
.	O
The	O
multimodal	O
model	O
has	O
predicted	O
a	O
closer	O
score	O
(	O
-0.002	O
)	O
to	O
the	O
gold	O
MQM	O
score	O
(	O
0.167	O
)	O
than	O
the	O
baseline	O
model	O
(	O
-0.248	O
)	O
.	O
The	O
French	O
translation	O
is	O
poor	O
(	O
cumulative	O
-	O
split	O
is	O
,	O
for	O
instance	O
,	O
not	O
translated	O
)	O
as	O
the	O
low	O
gold	O
MQM	O
score	O
shows	O
.	O
However	O
,	O
the	O
(	O
main	O
)	O
word	O
stopwatch	O
is	O
correctly	O
translated	O
as	O
chronomètre	O
in	O
French	O
.	O
Since	O
the	O
associated	O
picture	O
indeed	O
represents	O
a	O
stopwatch	O
,	O
one	O
explanation	O
for	O
this	O
improvement	O
could	O
be	O
that	O
the	O
multimodal	O
model	O
may	O
have	O
rewarded	O
this	O
correct	O
and	O
important	O
part	O
of	O
the	O
translation	O
.	O

Le	O
chronomètre	O
A601X	O
dispose	O
calendrier	O
cumulative	O
-	O
split	O
.	O
gold	O
MQM	O
score	O
0.167	O
BiRNN	O
-0.248	O
BiRNN+Vis	O
-	O
embed	O
-	O
mult2	O
-0.002	O
Table	O
3	O
:	O
Example	O
of	O
performance	O
of	O
sentence	O
-	O
level	O
multimodal	O
QE	O
.	O
Compared	O
to	O
the	O
baseline	O
prediction	O
(	O
BiRNN	O
)	O
,	O
the	O
prediction	O
from	O
the	O
best	O
multimodal	O
model	O
(	O
BiRNN+Vis	O
-	O
embed	O
-	O
mult2	O
)	O
is	O
closer	O
to	O
the	O
gold	O
MQM	O
score	O
.	O
This	O
could	O
be	O
because	O
the	O
word	O
stopwatch	O
is	O
correctly	O
translated	O
as	O
chronomètre	O
in	O
French	O
,	O
and	O
the	O
additional	O
visual	O
feature	O
confirms	O
it	O
.	O
This	O
could	O
lead	O
to	O
an	O
increase	O
in	O
the	O
predicted	O
score	O
to	O
reward	O
the	O
correct	O
part	O
,	O
despite	O
the	O
poor	O
translation	O
(	O
extracted	O
from	O
the	O
Amazon	O
Reviews	O
Dataset	O
of	O
McAuley	O
et	O
al	O
.	O
,	O
2015	O
)	O
.	O

Document	O
-	O
level	O
MQE	O

Table	O
4	O
presents	O
the	O
results	O
for	O
the	O
documentlevel	O
feature	O
-	O
based	O
and	O
BiRNN	O
neural	O
QE	O
models	O
.	O
1	O
The	O
first	O
section	O
shows	O
the	O
official	O
models	O
from	O
the	O
WMT'18	O
QE	O
Task	O
4	O
report	O
(	O
Specia	O
et	O
al	O
.	O
,	O
2018a	O
)	O
.	O
The	O
neural	O
-	O
based	O
approach	O
SHEF	O
-	O
PT	O
is	O
the	O
winning	O
submission	O
,	O
outperforming	O
another	O
neural	O
-	O
based	O
approach	O
(	O
SHEF	O
-	O
mtl	O
-	O
bRNN	O
)	O
.	O
For	O
our	O
BiRNN	O
models	O
(	O
second	O
section	O
)	O
,	O
BiRNN+Visembed	O
-	O
conc	O
performs	O
only	O
slightly	O
better	O
than	O
the	O
monomodal	O
baseline	O
.	O
For	O
the	O
feature	O
-	O
based	O
models	O
(	O
third	O
section	O
)	O
,	O
on	O
the	O
other	O
hand	O
,	O
the	O
baseline	O
monomodal	O
QuEst++	O
is	O
outperformed	O
by	O
various	O
multimodal	O
variants	O
by	O
a	O
large	O
margin	O
,	O
with	O
the	O
one	O
with	O
two	O
principal	O
components	O
(	O
QuEst+Vis-2	O
)	O
performing	O
the	O
best	O
.	O
The	O
more	O
PCA	O
components	O
kept	O
,	O
the	O
worse	O
the	O
results	O
(	O
see	O
Appendix	O
for	O
full	O
set	O
of	O
results	O
)	O
.	O
Figure	O
3	O
shows	O
the	O
Williams	O
significance	O
test	O
for	O
document	O
-	O
level	O
QuEst++	O
on	O
the	O
WMT'18	O
dataset	O
.	O

As	O
we	O
can	O
see	O
,	O
QuEst+Vis-2	O
model	O
outperforms	O
the	O
baseline	O
with	O
p	O
-	O
value	O
=	O
0.002	O
.	O
Thus	O
,	O
visual	O
features	O
significantly	O
improve	O
the	O
performance	O
of	O
featurebased	O
QE	O
systems	O
compared	O
to	O
the	O
monomodal	O
QE	O
counterparts	O
.	O

We	O
introduced	O
Multimodal	O
Quality	O
Estimation	O
for	O
Machine	O
Translation	O
,	O
where	O
an	O
external	O
modality	O
-visual	O
information	O
-is	O
incorporated	O
to	O
featurebased	O
and	O
neural	O
-	O
based	O
QE	O
approaches	O
,	O
on	O
sentence	O
and	O
document	O
levels	O
.	O
The	O
use	O
of	O
visual	O
features	O
extracted	O
from	O
images	O
has	O
led	O
to	O
significant	O
improvements	O
in	O
the	O
results	O
of	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
QE	O
approaches	O
,	O
especially	O
at	O
sentence	O
level	O
.	O

The	O
version	O
of	O
deepQuest	O
for	O
multimodal	O
QE	O
and	O
scripts	O
to	O
convert	O
document	O
into	O
sentencelevel	O
data	O
are	O
available	O
on	O
https://github.com/	O
sheffieldnlp	O
/	O
deepQuest	O
.	O

A	O
Appendix	O
PCA	O
analysis	O
Figure	O
4	O
shows	O
an	O
almost	O
linear	O
relationship	O
between	O
the	O
number	O
of	O
principal	O
components	O
and	O
the	O
explained	O
variance	O
of	O
the	O
PCA	O
(	O
see	O
Section	O
3.1	O
)	O
,	O
i.e.	O
the	O
higher	O
the	O
number	O
of	O
principal	O
components	O
,	O
the	O
larger	O
the	O
explained	O
variance	O
.	O
Therefore	O
,	O
we	O
experimented	O
with	O
various	O
numbers	O
of	O
components	O
up	O
to	O
15	O
(	O
1	O
,	O
2	O
,	O
3	O
,	O
5	O
,	O
10	O
,	O
and	O
15	O
)	O
on	O
the	O
development	O
set	O
to	O
find	O
the	O
best	O
settings	O
for	O
quality	O
prediction	O
.	O
Complete	O
results	O
Tables	O
5	O
and	O
6	O
present	O
the	O
full	O
set	O
of	O
results	O
of	O
our	O
experiments	O
on	O
document	O
and	O
sentence	O
-	O
level	O
multimodal	O
QE	O
on	O
our	O
main	O
test	O
set	O
,	O
the	O
WMT'18	O
test	O
set	O
.	O
These	O
are	O
a	O
super	O
-	O
set	O
of	O
the	O
results	O
presented	O
in	O
the	O
main	O
paper	O
but	O
include	O
all	O
combinations	O
of	O
multimodality	O
integration	O
and	O
fusion	O
strategies	O
for	O
sentence	O
-	O
level	O
prediction	O
,	O
as	O
well	O
as	O
different	O
numbers	O
of	O
principal	O
components	O
kept	O
for	O
document	O
-	O
level	O
QuEst	O
prediction	O
models	O
.	O

Additional	O
test	O
set	O
Tables	O
7	O
and	O
8	O
present	O
the	O
full	O
set	O
of	O
results	O
of	O
our	O
experiments	O
on	O
the	O
WMT'19	O
Task	O
2	O
test	O
set	O
on	O
document	O
and	O
sentencelevel	O
multimodal	O
QE	O
,	O
respectively	O
.	O
This	O
was	O
the	O
follow	O
-	O
up	O
edition	O
of	O
the	O
WMT'18	O
Task	O
4	O
,	O
where	O
the	O
same	O
training	O
set	O
is	O
used	O
,	O
but	O
a	O
new	O
test	O
set	O
is	O
released	O
.	O

For	O
sentence	O
-	O
level	O
,	O
we	O
observe	O
on	O
the	O
one	O
hand	O
quite	O
significant	O
improvements	O
with	O
a	O
gain	O
of	O
almost	O
8	O
points	O
in	O
Pearson	O
's	O
r	O
over	O
BiRNN	O
,	O
our	O
monomodal	O
baseline	O
without	O
pre	O
-	O
trained	O
word	O
embedding	O
.	O
multimodal	O
variants	O
achieve	O
better	O
performance	O
compared	O
to	O
the	O
monomodal	O
BiRNN	O
baseline	O
,	O
with	O
a	O
peak	O
when	O
the	O
visual	O
features	O
are	O
fused	O
with	O
the	O
word	O
embedding	O
representations	O
by	O
elementwise	O
multiplication	O
.	O
On	O
the	O
other	O
hand	O
,	O
we	O
do	O
not	O
observe	O
any	O
gain	O
in	O
using	O
visual	O
features	O
on	O
the	O
WMT'19	O
test	O
set	O
compared	O
to	O
our	O
monomodal	O
baseline	O
with	O
pre	O
-	O
trained	O
word	O
-	O
embedding	O
(	O
BERT	O
-	O
BiRNN	O
)	O
.	O
Here	O
that	O
the	O
BERT	O
-	O
BiRNN	O
baseline	O
model	O
already	O
performs	O
very	O
well	O
.	O
According	O
to	O
the	O
task	O
organisers	O
,	O
the	O
mean	O
MQM	O
value	O
on	O
the	O
WMT'19	O
test	O
set	O
is	O
higher	O
than	O
on	O
the	O
WMT'18	O
test	O
set	O
,	O
but	O
actually	O
closer	O
to	O
the	O
training	O
data	O
(	O
Fonseca	O

.	O
We	O
therefore	O
hypothesise	O
here	O
that	O
the	O
highly	O
dimensional	O
and	O
contextualised	O
word	O
-	O
level	O
representations	O
from	O
BERT	O
are	O
already	O
enough	O
and	O
do	O
not	O
benefit	O
from	O
the	O
extra	O
information	O
provided	O
by	O
the	O
visual	O
features	O
.	O

The	O
SOFC	O
-	O
Exp	O
Corpus	O
and	O
Neural	O
Approaches	O
to	O
Information	O
Extraction	O
in	O
the	O
Materials	O
Science	O
Domain	O

This	O
paper	O
presents	O
a	O
new	O
challenging	O
information	O
extraction	O
task	O
in	O
the	O
domain	O
of	O
materials	O
science	O
.	O
We	O
develop	O
an	O
annotation	O
scheme	O
for	O
marking	O
information	O
on	O
experiments	O
related	O
to	O
solid	O
oxide	O
fuel	O
cells	O
in	O
scientific	O
publications	O
,	O
such	O
as	O
involved	O
materials	O
and	O
measurement	O
conditions	O
.	O
With	O
this	O
paper	O
,	O
we	O
publish	O
our	O
annotation	O
guidelines	O
,	O
as	O
well	O
as	O
our	O
SOFC	O
-	O
Exp	O
corpus	O
consisting	O
of	O
45	O
openaccess	O
scholarly	O
articles	O
annotated	O
by	O
domain	O
experts	O
.	O
A	O
corpus	O
and	O
an	O
inter	O
-	O
annotator	O
agreement	O
study	O
demonstrate	O
the	O
complexity	O
of	O
the	O
suggested	O
named	O
entity	O
recognition	O
and	O
slot	O
filling	O
tasks	O
as	O
well	O
as	O
high	O
annotation	O
quality	O
.	O
We	O
also	O
present	O
strong	O
neural	O
-	O
network	O
based	O
models	O
for	O
a	O
variety	O
of	O
tasks	O
that	O
can	O
be	O
addressed	O
on	O
the	O
basis	O
of	O
our	O
new	O
data	O
set	O
.	O
On	O
all	O
tasks	O
,	O
using	O
BERT	O
embeddings	O
leads	O
to	O
large	O
performance	O
gains	O
,	O
but	O
with	O
increasing	O
task	O
complexity	O
,	O
adding	O
a	O
recurrent	O
neural	O
network	O
on	O
top	O
seems	O
beneficial	O
.	O
Our	O
models	O
will	O
serve	O
as	O
competitive	O
baselines	O
in	O
future	O
work	O
,	O
and	O
analysis	O
of	O
their	O
performance	O
highlights	O
difficult	O
cases	O
when	O
modeling	O
the	O
data	O
and	O
suggests	O
promising	O
research	O
directions	O
.	O

The	O
design	O
of	O
new	O
experiments	O
in	O
scientific	O
domains	O
heavily	O
depends	O
on	O
domain	O
knowledge	O
as	O
well	O
as	O
on	O
previous	O
studies	O
and	O
their	O
findings	O
.	O
However	O
,	O
the	O
amount	O
of	O
publications	O
available	O
is	O
typically	O
very	O
large	O
,	O
making	O
it	O
hard	O
or	O
even	O
impossible	O
to	O
keep	O
track	O
of	O
all	O
experiments	O
conducted	O
for	O
a	O
particular	O
research	O
question	O
.	O
Since	O
scientific	O
experiments	O
are	O
often	O
time	O
-	O
consuming	O
and	O
expensive	O
,	O
effective	O
knowledge	O
base	O
population	O
methods	O
for	O
finding	O
promising	O
settings	O
based	O
on	O
the	O
published	O
research	O
would	O
be	O
of	O
great	O
value	O
(	O
e.g.	O
,	O
Auer	O
et	O
al	O
.	O
,	O
2018;Manica	O
et	O
al	O
.	O
,	O
2019;Mrdjenovich	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
While	O
such	O
real	O
-	O
life	O
information	O
extraction	O
tasks	O
have	O
received	O
consid-	O
erable	O
attention	O
in	O
the	O
biomedical	O
domain	O
(	O
e.g.	O
,	O
Cohen	O
et	O
al	O
.	O
,	O
2017;Demner	O
-	O
Fushman	O
et	O
al	O
.	O
,	O
2018	O
,	O
there	O
has	O
been	O
little	O
work	O
in	O
other	O
domains	O
(	O
Nastase	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
including	O
materials	O
science	O
(	O
with	O
the	O
notable	O
exception	O
of	O
the	O
work	O
by	O
Mysore	O
et	O
al	O
.	O
,	O
2017Mysore	O
et	O
al	O
.	O
,	O
,	O
2019	O
.	O

In	O
this	O
paper	O
,	O
we	O
introduce	O
a	O
new	O
information	O
extraction	O
use	O
case	O
from	O
the	O
materials	O
science	O
domain	O
and	O
propose	O
a	O
series	O
of	O
new	O
challenging	O
information	O
extraction	O
tasks	O
.	O
We	O
target	O
publications	O
about	O
solid	O
oxide	O
fuel	O
cells	O
(	O
SOFCs	O
)	O
in	O
which	O
the	O
interdependence	O
between	O
chosen	O
materials	O
,	O
measurement	O
conditions	O
and	O
performance	O
is	O
complex	O
(	O
see	O
Figure	O
1	O
)	O
.	O
For	O
making	O
progress	O
within	O
natural	O
language	O
processing	O
(	O
NLP	O
)	O
,	O
the	O
genre	O
-	O
domain	O
combination	O
presents	O
interesting	O
challenges	O
and	O
characteristics	O
,	O
e.g.	O
,	O
domain	O
-	O
specific	O
tokens	O
such	O
as	O
material	O
names	O
and	O
chemical	O
formulas	O
.	O

The	O
task	O
of	O
finding	O
experiment	O
-	O
specific	O
information	O
can	O
be	O
modeled	O
as	O
a	O
retrieval	O
task	O
(	O
i.e.	O
,	O
finding	O
relevant	O
information	O
in	O
documents	O
)	O
and	O
at	O
the	O
same	O
time	O
as	O
a	O
semantic	O
-	O
role	O
-	O
labeling	O
task	O
(	O
i.e.	O
,	O
identifying	O
the	O
slot	O
fillers	O
)	O
.	O
We	O
identify	O
three	O
sub	O
-	O
tasks	O
:	O

(	O
1	O
)	O
identifying	O
sentences	O
describing	O
relevant	O
experiments	O
,	O
(	O
2	O
)	O
identifying	O
mentions	O
of	O
materials	O
,	O
values	O
,	O
and	O
devices	O
,	O
and	O
(	O
3	O
)	O
recognizing	O
mentions	O
of	O
slots	O
and	O
their	O
values	O
related	O
to	O
these	O
experiments	O
.	O
We	O
propose	O
and	O
compare	O
several	O
machine	O
learning	O
methods	O
for	O
the	O
different	O
sub	O
-	O
tasks	O
,	O
including	O
bidirectional	O
long	O
-	O
short	O
term	O
memory	O
(	O
BiLSTM	O
)	O
networks	O
and	O
BERT	O
-	O
based	O
models	O
.	O
In	O
our	O
results	O
,	O
BERT	O
-	O
based	O
models	O
show	O
superior	O
performance	O
.	O
However	O
,	O
with	O
increasing	O
complexity	O
of	O
the	O
task	O
,	O
it	O
is	O
beneficial	O
to	O
combine	O
the	O
two	O
approaches	O
.	O

With	O
the	O
aim	O
of	O
fostering	O
research	O
on	O
challenging	O
information	O
extraction	O
tasks	O
in	O
the	O
scientific	O
domain	O
,	O
we	O
target	O
the	O
domain	O
of	O
SOFC	O
-	O
related	O
experiments	O
as	O
a	O
starting	O
point	O
.	O
Our	O
findings	O
based	O
on	O
this	O
sample	O
use	O
case	O
are	O
transferable	O
to	O
similar	O
experimental	O
domains	O
,	O
which	O
we	O
illustrate	O
by	O
applying	O
our	O
best	O
model	O
configurations	O
to	O
a	O
previously	O
existing	O
related	O
corpus	O
(	O
Mysore	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
achieving	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
results	O
.	O

•	O
We	O
provide	O
a	O
new	O
corpus	O
of	O
45	O
materialsscience	O
publications	O
in	O
the	O
research	O
area	O
of	O
SOFCs	O
,	O
manually	O
annotated	O
by	O
domain	O
experts	O
for	O
information	O
on	O
experimental	O
settings	O
and	O
results	O
(	O
Section	O
4	O
)	O
.	O
Our	O
corpus	O
is	O
publicly	O
available	O
.	O
1	O
Our	O
inter	O
-	O
annotator	O
agreement	O
study	O
provides	O
evidence	O
for	O
high	O
annotation	O
quality	O
(	O
Section	O
5	O
)	O
.	O

Information	O
extraction	O
for	O
scientific	O
publications	O
.	O
Recently	O
,	O
several	O
studies	O
addressed	O
information	O
extraction	O
and	O
knowledge	O
base	O
construction	O
in	O
the	O
scientific	O
domain	O
(	O
Augenstein	O
et	O
al	O
.	O
,	O
2017;Luan	O
et	O
al	O
.	O
,	O
2018;Jiang	O
et	O
al	O
.	O
,	O
2019;Buscaldi	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
We	O
also	O
aim	O
at	O
knowledge	O
base	O
construction	O
but	O
target	O
publications	O
about	O
materials	O
science	O
experiments	O
,	O
a	O
domain	O
understudied	O
in	O
NLP	O
to	O
date	O
.	O
Information	O
extraction	O
for	O
materials	O
science	O
.	O
The	O
work	O
closest	O
to	O
ours	O
is	O
the	O
one	O
of	O
Mysore	O
et	O
al	O
.	O
(	O
2019	O
)	O
also	O
retrieve	O
synthesis	O
procedures	O
and	O
extract	O
recipes	O
,	O
though	O
with	O
a	O
coarser	O
-	O
grained	O
label	O
set	O
,	O
focusing	O
on	O
different	O
synthesis	O
operation	O
types	O
.	O
create	O
a	O
dataset	O
for	O
named	O
entity	O
recognition	O
on	O
abstracts	O
of	O
materials	O
science	O
publications	O
.	O
In	O
contrast	O
to	O
our	O
work	O
,	O
their	O
label	O
set	O
(	O
e.g.	O
,	O
Material	O
,	O
Application	O
,	O
Property	O
)	O
is	O
targeted	O
to	O
document	O
indexing	O
rather	O
than	O
information	O
extraction	O
.	O
A	O
notable	O
difference	O
to	O
our	O
work	O
is	O
that	O
we	O
perform	O
full	O
-	O
text	O
annotation	O
while	O
the	O
aforementioned	O
approaches	O
annotate	O
a	O
pre	O
-	O
selected	O
set	O
of	O
paragraphs	O
(	O
see	O
also	O
.	O
Mysore	O
et	O
al	O
.	O
(	O
2017	O
)	O
apply	O
the	O
generative	O
model	O
of	O
Kiddon	O
et	O
al	O
.	O
(	O
2015	O
)	O
to	O
induce	O
action	O
graphs	O
for	O
synthesis	O
procedures	O
of	O
materials	O
from	O
text	O
.	O
In	O
Section	O
7.1	O
,	O
we	O
implement	O
a	O
similar	O
entity	O
extraction	O
system	O
and	O
also	O
apply	O
our	O
algorithms	O
to	O
the	O
dataset	O
of	O
Mysore	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O
train	O
word2vec	O
(	O
Mikolov	O
et	O
al	O
.	O
,	O
2013	O
)	O
embeddings	O
on	O
materials	O
science	O
publications	O
and	O
show	O
that	O
they	O
can	O
be	O
used	O
for	O
recommending	O
materials	O
for	O
functional	O
applications	O
.	O
Other	O
works	O
adapt	O
the	O
BERT	O
model	O
to	O
clinical	O
and	O
biomedical	O
domains	O
(	O
Alsentzer	O
et	O
al	O
.	O
,	O
2019;Sun	O
and	O
Yang	O
,	O
2019	O
)	O
,	O
or	O
generally	O
to	O
scientific	O
text	O
(	O
Beltagy	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O

Neural	O
entity	O
tagging	O
and	O
slot	O
filling	O
.	O
The	O
neural	O
-	O
network	O
based	O
models	O
we	O
use	O
for	O
entity	O
tagging	O
and	O
slot	O
filling	O
bear	O
similarity	O
to	O
state	O
-	O
ofthe	O
-	O
art	O
models	O
for	O
named	O
entity	O
recognition	O
(	O
e.g.	O
,	O
Huang	O
et	O
al	O
.	O
,	O
2015;Lample	O
et	O
al	O
.	O
,	O
2016;Panchendrarajan	O
and	O
Amaresan	O
,	O
2018;Lange	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Other	O
related	O
work	O
exists	O
in	O
the	O
area	O
of	O
semantic	O
role	O
labeling	O
(	O
e.g.	O
,	O
Roth	O
and	O
Lapata	O
,	O
2015;Kshirsagar	O
et	O
al	O
.	O
,	O
2015;Hartmann	O
et	O
al	O
.	O
,	O
2017;Adel	O
et	O
al	O
.	O
,	O
2018;Swayamdipta	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O

In	O
this	O
section	O
,	O
we	O
describe	O
our	O
annotation	O
scheme	O
and	O
guidelines	O
for	O
marking	O
information	O
on	O
SOFCrelated	O
experiments	O
in	O
scientific	O
publications	O
.	O

We	O
treat	O
the	O
annotation	O
task	O
as	O
identifying	O
instances	O
of	O
a	O
semantic	O
frame	O
(	O
Fillmore	O
,	O
1976	O
)	O
that	O
represents	O
SOFC	O
-	O
related	O
experiments	O
.	O
We	O
include	O
(	O
1	O
)	O
cases	O
that	O
introduce	O
novel	O
content	O
;	O
(	O
2	O
)	O
descriptions	O
of	O
specific	O
previous	O
work	O
;	O
(	O
3	O
)	O
general	O
knowledge	O
that	O
one	O
could	O
find	O
in	O
a	O
textbook	O
or	O
survey	O
;	O
and	O
also	O
(	O
4	O
)	O
suggestions	O
for	O
future	O
work	O
.	O

The	O
above	O
two	O
steps	O
of	O
recognizing	O
relevant	O
sentences	O
and	O
marking	O
coarse	O
-	O
grained	O
entity	O
types	O
are	O
in	O
general	O
applicable	O
to	O
a	O
wide	O
range	O
of	O
experiment	O
types	O
within	O
the	O
materials	O
science	O
domain	O
.	O
We	O
now	O
define	O
a	O
set	O
of	O
slot	O
types	O
particular	O
to	O
experiments	O
on	O
SOFCs	O
.	O
During	O
annotation	O
,	O
we	O
mark	O
these	O
slot	O
types	O
as	O
links	O
between	O
the	O
experimentevoking	O
phrase	O
and	O
the	O
respective	O
slot	O
filler	O
(	O
entity	O
mention	O
)	O
,	O
see	O
Figure	O
1	O
.	O
As	O
a	O
result	O
,	O
experiment	O
frames	O
are	O
represented	O
by	O
graphs	O
rooted	O
in	O
the	O
node	O
corresponding	O
to	O
the	O
frame	O
-	O
evoking	O
element	O
.	O

Our	O
annotation	O
scheme	O
comprises	O
16	O
slot	O
types	O
relevant	O
for	O
SOFC	O
experiments	O
.	O
Here	O
we	O
explain	O
a	O
few	O
of	O
these	O
types	O
for	O
illustration	O
.	O
A	O
full	O
list	O
of	O
these	O
slot	O
types	O
can	O
be	O
found	O
in	O
Supplementary	O
Material	O
Table	O
11	O
;	O
detailed	O
explanations	O
are	O
given	O
in	O
the	O
annotation	O
guidelines	O
published	O
along	O
with	O
our	O
corpus	O
.	O
PowerDensity	O
,	O
Resistance	O
,	O
WorkingTemperature	O
:	O
These	O
slots	O
are	O
generally	O
filled	O
by	O
mentions	O
of	O
type	O
VALUE	O
,	O
i.e.	O
,	O
a	O
numerical	O
value	O
plus	O
a	O
unit	O
.	O
Our	O
annotation	O
guidelines	O
give	O
examples	O
for	O
relevant	O
units	O
and	O
describe	O
special	O
cases	O
.	O
This	O
enables	O
any	O
materials	O
scientist	O
,	O
even	O
if	O
he	O
/	O
she	O
is	O
not	O
an	O
expert	O
on	O
SOFCs	O
,	O
to	O
easily	O
understand	O
and	O
apply	O
our	O
annotation	O
guidelines	O
.	O

SOFC	O
-	O
Exp	O
Corpus	O
.	O
Our	O
corpus	O
consists	O
of	O
45	O

open	O
-	O
access	O
scientific	O
publications	O
about	O
SOFCs	O
and	O
related	O
research	O
,	O
annotated	O
by	O
domain	O
experts	O
.	O

Task	O
definitions	O
.	O
Our	O
rich	O
graph	O
-	O
based	O
annotation	O
scheme	O
allows	O
for	O
a	O
number	O
of	O
information	O
extraction	O
tasks	O
.	O
In	O
the	O
scope	O
of	O
this	O
paper	O
,	O
we	O
address	O
the	O
following	O
steps	O
of	O
(	O
1	O
)	O
identifying	O
sentences	O
that	O
describe	O
SOFC	O
-	O
related	O
experiments	O
,	O
(	O
2	O
)	O
recognizing	O
and	O
typing	O
relevant	O
named	O
entities	O
,	O
and	O

We	O
here	O
present	O
the	O
results	O
of	O
our	O
inter	O
-	O
annotator	O
agreement	O
study	O
,	O
which	O
we	O
perform	O
in	O
order	O
to	O
estimate	O
the	O
degree	O
of	O
reproducibility	O
of	O
our	O
corpus	O
and	O
to	O
put	O
automatic	O
modeling	O
performance	O
into	O
perspective	O
.	O
Six	O
documents	O
(	O
973	O
sentences	O
)	O
have	O
been	O
annotated	O
independently	O
both	O
by	O
our	O
primary	O
annotator	O
,	O
a	O
graduate	O
student	O
of	O
materials	O
science	O
,	O
and	O
a	O
second	O
annotator	O
,	O
who	O
holds	O
a	O
Ph.D.	O
in	O
physics	O
and	O
is	O
active	O
in	O
the	O
field	O
of	O
materials	O
science	O
.	O
The	O
label	O
distribution	O
in	O
this	O
subset	O
is	O
similar	O
to	O
the	O
one	O
of	O
our	O
overall	O
corpus	O
,	O
with	O
each	O
annotator	O
choosing	O
EXPERIMENT	O
about	O
11.8	O
%	O
of	O
the	O
time	O
.	O
Identification	O
of	O
experiment	O
-	O
describing	O
sentences	O
.	O
Agreement	O
on	O
our	O
first	O
task	O
,	O
judging	O
whether	O
a	O
sentence	O
contains	O
relevant	O
experimental	O
information	O
,	O
is	O
0.75	O
in	O
terms	O
of	O
Cohen	O
's	O
κ	O
(	O
Cohen	O
,	O
1968	O
)	O
,	O
indicating	O
substantial	O
agreement	O
according	O
to	O
Landis	O
and	O
Koch	O
(	O
1977	O
)	O
.	O
The	O
observed	O
agreement	O
,	O
corresponding	O
to	O
accuracy	O
,	O
is	O
94.9	O
%	O
;	O
expected	O
agreement	O
amounts	O
to	O
79.2	O
%	O
.	O
Table	O
2	O
shows	O
precision	O
,	O
recall	O
and	O
F1	O
for	O
the	O
doubly	O
-	O
annotated	O
subset	O
,	O
treating	O
one	O
annotator	O
as	O
the	O
gold	O
standard	O
and	O
the	O
other	O
one	O
's	O
labels	O
as	O
predicted	O
.	O
Our	O
primary	O
annotator	O
identifies	O
119	O
out	O
of	O
973	O
sentences	O
as	O
experiment	O
-	O
describing	O
,	O
our	O
secondary	O
annotator	O
111	O
sentences	O
,	O
with	O
an	O
overlap	O
of	O
90	O
sentences	O
.	O
These	O
statistics	O
are	O
helpful	O
to	O
gain	O
further	O
intuition	O
of	O
how	O
well	O
a	O
human	O
can	O
reproduce	O
another	O
annotator	O
's	O
labels	O
and	O
can	O
also	O
be	O
considered	O
an	O
upper	O
bound	O
for	O
system	O
performance	O
.	O

Entity	O
mention	O
detection	O
and	O
type	O
assignment	O
.	O

As	O
mentioned	O
above	O
,	O
relevant	O
entity	O
mentions	O
and	O
their	O
types	O
are	O
only	O
annotated	O
for	O
sentences	O
containing	O
experiment	O
information	O
and	O
neighboring	O
sentences	O
.	O
Therefore	O
,	O
we	O
here	O
compute	O
agreement	O
on	O
the	O
detection	O
of	O
entity	O
mention	O
and	O
type	O
assignment	O
on	O
the	O
subset	O
of	O
90	O
sentences	O
that	O
both	O
annotators	O
considered	O
as	O
containing	O
experimental	O
information	O
.	O
We	O
again	O
look	O
at	O
precision	O
and	O
recall	O
of	O
the	O
annotators	O
versus	O
each	O
other	O
,	O
see	O
Table	O
3	O
.	O

The	O
high	O
precision	O
indicates	O
that	O
our	O
secondary	O
annotator	O
marks	O
essentially	O
the	O
same	O
mentions	O
as	O
our	O
primary	O
annotator	O
,	O
but	O
recall	O
suggests	O
a	O
few	O
missing	O
cases	O
.	O
The	O
difference	O
in	O
marking	O
EXPERI	O
-	O
MENT	O
can	O
be	O
explained	O
by	O
the	O
fact	O
that	O
the	O
primary	O
annotator	O
sometimes	O
marks	O
several	O
verbs	O
per	O
sentence	O
as	O
experiment	O
-	O
evoking	O
elements	O
,	O
connecting	O
them	O
with	O
same	O
exp	O
or	O
exp	O
variation	O
,	O
while	O
the	O
secondary	O
annotator	O
links	O
the	O
mentions	O
of	O
relevant	O
slots	O
to	O
the	O
first	O
experiment	O
-	O
evoking	O
element	O
(	O
see	O
also	O
Supplementary	O
Material	O
Section	O
B	O
)	O
.	O
Overall	O
,	O
the	O
high	O
agreement	O
between	O
domain	O
expert	O
annotators	O
indicates	O
high	O
data	O
quality	O
.	O
Identifying	O
experiment	O
slot	O
fillers	O
.	O
We	O
compute	O
agreement	O
on	O
the	O
task	O
of	O
identifying	O
the	O
slots	O
of	O
an	O
experiment	O
frame	O
filled	O
by	O
the	O
mentions	O
in	O
a	O
sentence	O
on	O
the	O
subset	O
of	O
sentences	O
that	O
both	O
annotators	O
marked	O
as	O
experiment	O
-	O
describing	O
.	O
Slot	O
fillers	O
are	O
the	O
dependents	O
of	O
the	O
respective	O
edges	O
starting	O
at	O
the	O
experiment	O
-	O
evoking	O
element	O
.	O
Table	O
4	O
shows	O
F1	O
scores	O
for	O
the	O
most	O
frequent	O
ones	O
among	O
those	O
categories	O
.	O
See	O
Supplementary	O
Material	O
Section	O
C	O
for	O
all	O
slot	O
types	O
.	O
Overall	O
,	O
our	O
agreement	O
study	O
provides	O
support	O
for	O
the	O
high	O
quality	O
of	O
our	O
annotation	O
scheme	O
and	O
validates	O
the	O
annotated	O
dataset	O
.	O

Experiment	O
detection	O
.	O
The	O
task	O
of	O
experiment	O
detection	O
can	O
be	O
modeled	O
as	O
a	O
binary	O
sentence	O
classification	O
problem	O
.	O
It	O
can	O
also	O
be	O
conceived	O
as	O
a	O
retrieval	O
task	O
,	O
selecting	O
sentences	O
as	O
candidates	O
for	O
experiment	O
frame	O
extraction	O
.	O
We	O
implement	O
a	O
bidirectional	O
long	O
short	O
-	O
term	O
memory	O
(	O
BiLSTM	O
)	O
model	O
with	O
attention	O
for	O
the	O
task	O
of	O
experiment	O
sentence	O
detection	O
.	O
Each	O
input	O
token	O
is	O
represented	O
by	O
a	O
concatenation	O
of	O
several	O
pretrained	O
word	O
embeddings	O
,	O
each	O
of	O
which	O
is	O
fine	O
-	O
tuned	O
during	O
training	O
.	O
We	O
use	O
the	O
Google	O
News	O
word2vec	O
embeddings	O
(	O
Mikolov	O
et	O
al	O
.	O
,	O
2013	O
)	O
,	O
domain	O
-	O
specific	O
word2vec	O
embeddings	O
(	O
mat2vec	O
,	O
,	O
see	O
also	O
Section	O
2	O
)	O
,	O
subword	O
embeddings	O
based	O
on	O
byte	O
-	O
pair	O
encoding	O
(	O
bpe	O
,	O
Heinzerling	O
and	O
Strube	O
,	O
2018	O
)	O
,	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
and	O
SciBERT	O
(	O
Beltagy	O
et	O
al	O
.	O
,	O
2019	O
)	O
embeddings	O
.	O
For	O
BERT	O
and	O
SciBERT	O
,	O
we	O
take	O
the	O
embeddings	O
of	O
the	O
first	O
word	O
piece	O
as	O
token	O
representation	O
.	O
The	O
embeddings	O
are	O
fed	O
into	O
a	O
BiLSTM	O
model	O
followed	O
by	O
an	O
attention	O
layer	O
that	O
computes	O
a	O
vector	O
for	O
the	O
whole	O
sentence	O
.	O
Finally	O
,	O
a	O
softmax	O
layer	O
decides	O
whether	O
the	O
sentence	O
contains	O
an	O
experiment	O
.	O

In	O
addition	O
,	O
we	O
fine	O
-	O
tune	O
the	O
original	O
(	O
uncased	O
)	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
as	O
well	O
as	O
SciBERT	O
(	O
Beltagy	O
et	O
al	O
.	O
,	O
2019	O
)	O
models	O
on	O
our	O
dataset	O
.	O
Sci	O
-	O
BERT	O
was	O
trained	O
on	O
a	O
large	O
corpus	O
of	O
scientific	O
text	O
.	O
We	O
use	O
the	O
implementation	O
of	O
the	O
BERT	O
sentence	O
classifier	O
by	O
Wolf	O
et	O
al	O
.	O
(	O
2019	O
)	O
that	O
uses	O
the	O
CLS	O
token	O
of	O
BERT	O
as	O
input	O
to	O
the	O
classification	O
layer	O
.	O
5	O
Finally	O
,	O
we	O
compare	O
the	O
neural	O
network	O
models	O
with	O
traditional	O
classification	O
models	O
,	O
namely	O
a	O
support	O
vector	O
machine	O
(	O
SVM	O
)	O
and	O
a	O
logistic	O
regression	O
classifier	O
.	O
For	O
both	O
models	O
,	O
we	O
use	O
the	O
following	O
set	O
of	O
input	O
features	O
:	O
bag	O
-	O
of	O
-	O
words	O
vectors	O
indicating	O
which	O
1	O
-	O
to	O
4	O
-	O
grams	O
and	O
part	O
-	O
of	O
-	O
speech	O
tags	O
occur	O
in	O
the	O
sentence	O
.	O
6	O
Entity	O
mention	O
extraction	O
.	O
For	O
entity	O
and	O
concept	O
extraction	O
,	O
we	O
use	O
a	O
sequence	O
-	O
tagging	O
approach	O
similar	O
to	O
(	O
Huang	O
et	O
al	O
.	O
,	O
2015;Lample	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
namely	O
a	O
BiLSTM	O
model	O
.	O
We	O
use	O
the	O
same	O
input	O
representation	O
(	O
stacked	O
embeddings	O
)	O
as	O
above	O
,	O
which	O
are	O
fed	O
into	O
a	O
BiLSTM	O
.	O
The	O
subsequent	O
conditional	O
random	O
field	O
(	O
CRF	O
,	O
Lafferty	O
et	O
al	O
.	O
,	O
2001	O
)	O
output	O
layer	O
extracts	O
the	O
most	O
probable	O
label	O
sequence	O
.	O
To	O
cope	O
with	O
multi	O
-	O
token	O
entities	O
,	O
we	O
convert	O
the	O
labels	O
into	O
BIO	O
format	O
.	O

We	O
also	O
fine	O
-	O
tune	O
the	O
original	O
BERT	O
and	O
SciB	O
-	O
ERT	O
sequence	O
tagging	O
models	O
on	O
this	O
task	O
.	O
Since	O
we	O
use	O
BIO	O
labels	O
,	O
we	O
extend	O
it	O
with	O
a	O
CRF	O
output	O
layer	O
to	O
enable	O
it	O
to	O
correctly	O
label	O
multi	O
-	O
token	O
mentions	O
and	O
to	O
enable	O
it	O
to	O
learn	O
transition	O
scores	O
between	O
labels	O
.	O
As	O
a	O
non	O
-	O
neural	O
baseline	O
,	O
we	O
train	O
5	O
https://github.com/huggingface/	O
transformers	O
6	O
We	O
use	O
sklearn	O
,	O
https://scikit-learn.org	O
.	O

a	O
CRF	O
model	O
using	O
the	O
token	O
,	O
its	O
lemma	O
,	O
part	O
-	O
ofspeech	O
tag	O
and	O
mat2vec	O
embedding	O
as	O
features	O
.	O
7	O

Slot	O
filling	O
.	O
As	O
described	O
in	O
Section	O
4	O
,	O
we	O
approach	O
the	O
slot	O
filler	O
extraction	O
task	O
as	O
fine	O
-	O
grained	O
entity	O
-	O
typing	O
-	O
in	O
-	O
context	O
,	O
assuming	O
that	O
each	O
sentence	O
represents	O
a	O
single	O
experiment	O
frame	O
.	O
We	O
use	O
the	O
same	O
sequence	O
tagging	O
architectures	O
as	O
above	O
for	O
tagging	O
the	O
tokens	O
of	O
each	O
experimentdescribing	O
sentence	O
with	O
the	O
set	O
of	O
slot	O
types	O
(	O
see	O
Table	O
11	O
)	O
.	O
Future	O
work	O
may	O
contrast	O
this	O
sequence	O
tagging	O
baseline	O
with	O
graph	O
-	O
induction	O
based	O
frame	O
extraction	O
.	O

Hyperparameters	O
and	O
training	O
.	O
The	O
BiLSTM	O
models	O
are	O
trained	O
with	O
the	O
Adam	O
optimizer	O
(	O
Kingma	O
and	O
Ba	O
,	O
2015	O
)	O
with	O
a	O
learning	O
rate	O
of	O
1e-3	O
.	O
For	O
fine	O
-	O
tuning	O
the	O
original	O
BERT	O
models	O
,	O
we	O
follow	O
the	O
configuration	O
published	O
by	O
Wolf	O
et	O
al	O
.	O
(	O
2019	O
)	O
and	O
use	O
AdamW	O
(	O
Loshchilov	O
and	O
Hutter	O
,	O
2019	O
)	O
as	O
optimizer	O
and	O
a	O
learning	O
rate	O
of	O
4e-7	O
for	O
sentence	O
classification	O
and	O
1e-5	O
for	O
sequence	O
tagging	O
.	O
When	O
adding	O
BERT	O
tokens	O
to	O
the	O
BiLSTM	O
,	O
we	O
also	O
use	O
the	O
AdamW	O
optimizer	O
for	O
the	O
whole	O
model	O
and	O
learning	O
rates	O
of	O
4e-7	O
or	O
1e-5	O
for	O
the	O
BERT	O
part	O
and	O
1e-3	O
for	O
the	O
remainder	O
.	O
For	O
regularization	O
,	O
we	O
employ	O
early	O
stopping	O
on	O
the	O
development	O
set	O
.	O
We	O
use	O
a	O
stacked	O
BiLSTM	O
with	O
two	O
hidden	O
layers	O
and	O
500	O
hidden	O
units	O
for	O
all	O
tasks	O
with	O
the	O
exception	O
of	O
the	O
experiment	O
sentence	O
de-	O
tection	O
task	O
,	O
where	O
we	O
found	O
one	O
BiLSTM	O
layer	O
to	O
work	O
best	O
.	O
The	O
attention	O
layer	O
of	O
the	O
sentence	O
detection	O
model	O
has	O
a	O
hidden	O
size	O
of	O
100	O
.	O

Experiment	O
sentence	O
detection	O
.	O
Table	O
5	O
shows	O
our	O
results	O
on	O
the	O
detection	O
of	O
experimentdescribing	O
sentences	O
.	O
The	O
neural	O
models	O
with	O
bytepair	O
encoding	O
embeddings	O
or	O
BERT	O
clearly	O
outperform	O
the	O
SVM	O
and	O
logistic	O
regression	O
models	O
.	O
Within	O
the	O
neural	O
models	O
,	O
BERT	O
and	O
SciBERT	O
add	O
the	O
most	O
value	O
,	O
both	O
when	O
using	O
their	O
embeddings	O
as	O
another	O
input	O
to	O
the	O
BiLSTM	O
and	O
when	O
finetuning	O
the	O
original	O
BERT	O
models	O
.	O
Note	O
that	O
even	O
the	O
general	O
-	O
domain	O
BERT	O
is	O
strong	O
enough	O
to	O
cope	O
with	O
non	O
-	O
standard	O
domains	O
.	O
Nevertheless	O
,	O
models	O
based	O
on	O
SciBERT	O
outperform	O
BERT	O
-	O
based	O
models	O
,	O
indicating	O
that	O
in	O
-	O
domain	O
information	O
is	O
indeed	O
beneficial	O
.	O
For	O
performance	O
reasons	O
,	O
we	O
use	O
BERT	O
-	O
base	O
in	O
our	O
experiments	O
,	O
but	O
for	O
the	O
sake	O
of	O
completeness	O
,	O
we	O
also	O
run	O
BERT	O
-	O
large	O
for	O
the	O
task	O
of	O
detecting	O
experiment	O
sentences	O
.	O
Because	O
it	O
did	O
not	O
outperform	O
BERT	O
-	O
base	O
in	O
our	O
cross	O
-	O
validation	O
based	O
development	O
setting	O
,	O
we	O
did	O
not	O
further	O
experiment	O
with	O
BERT	O
-	O
large	O
.	O
However	O
,	O
we	O
found	O
that	O
it	O
resulted	O
in	O
the	O
best	O
F1	O
-	O
score	O
achieved	O
on	O
our	O
test	O
set	O
.	O
In	O
general	O
,	O
SciBERT	O
-	O
based	O
models	O
provide	O
very	O
good	O
performance	O
and	O
seem	O
most	O
robust	O
across	O
dev	O
and	O
test	O
sets	O
.	O
Overall	O
,	O
achieving	O
F1	O
-	O
scores	O
around	O
67.0	O
-	O
68.6	O
,	O
such	O
a	O
retrieval	O
model	O
may	O
already	O
be	O
useful	O
in	O
production	O
.	O
However	O
,	O
there	O
certainly	O
is	O
room	O
for	O
improvement	O
.	O
Entity	O
mention	O
extraction	O
.	O
Table	O
6	O
provides	O
our	O
results	O
on	O
entity	O
mention	O
detection	O
and	O
typing	O
.	O

Models	O
are	O
trained	O
and	O
results	O
are	O
reported	O
on	O
the	O
subset	O
of	O
sentences	O
marked	O
as	O
experimentdescribing	O
in	O
the	O
gold	O
standard	O
,	O
amounting	O
to	O
4,590	O
entity	O
mentions	O
in	O
total	O
.	O
9	O
The	O
CRF	O
baseline	O
achieves	O
comparable	O
or	O
better	O
results	O
than	O
the	O
Bi	O
-	O
LSTM	O
with	O
word2vec	O
and/or	O
mat2vec	O
embeddings	O
.	O
However	O
,	O
adding	O
subword	O
-	O
based	O
embeddings	O
(	O
bpe	O
and/or	O
BERT	O
)	O
significantly	O
increases	O
performance	O
of	O
the	O
BiLSTM	O
,	O
indicating	O
that	O
there	O
are	O
many	O
rare	O
words	O
.	O
Again	O
,	O
the	O
best	O
results	O
are	O
obtained	O
when	O
using	O
BERT	O
or	O
SciBERT	O
embeddings	O
or	O
when	O
using	O
the	O
original	O
SciBERT	O
model	O
.	O
It	O
is	O
relatively	O
easy	O
for	O
all	O
model	O
variants	O
to	O
recognize	O
VALUE	O
as	O
these	O
mentions	O
usually	O
consist	O
of	O
a	O
number	O
and	O
unit	O
which	O
the	O
model	O
can	O
easily	O
memorize	O
.	O
Recognizing	O
the	O
types	O
MATERIAL	O
and	O
DEVICE	O
,	O
in	O
contrast	O
,	O
is	O
harder	O
and	O
may	O
profit	O
from	O
using	O
gazetteer	O
-	O
based	O
extensions	O
.	O

Experiment	O
slot	O
filling	O
.	O
Table	O
7	O
shows	O
the	O
macro	O
-	O
average	O
F1	O
scores	O
for	O
our	O
different	O
models	O
on	O
the	O
slot	O
identification	O
task	O
.	O
10	O
As	O
for	O
entity	O
typing	O
,	O
we	O
train	O
and	O
evaluate	O
our	O
model	O
on	O
the	O
subset	O
of	O
sentences	O
marked	O
as	O
experiment	O
-	O
describing	O
,	O
which	O
contain	O
4,263	O
slot	O
instances	O
.	O
Again	O
,	O
the	O
CRF	O
baseline	O
outperforms	O
the	O
BiLSTM	O
when	O
using	O
only	O
mat2vec	O
and/or	O
word2vec	O
embeddings	O
.	O
The	O
addition	O
of	O
BERT	O
or	O
SciBERT	O
embeddings	O
improves	O
performance	O
.	O
However	O
,	O
on	O
this	O
task	O
,	O
the	O
BiLSTM	O
model	O
with	O
(	O
Sci)BERT	O
embeddings	O
outperforms	O
the	O
fine	O
-	O
tuned	O
original	O
(	O
Sci)BERT	O
model	O
.	O
Compared	O
to	O
the	O
other	O
two	O
tasks	O
,	O
this	O
task	O
requires	O
more	O
complex	O
reasoning	O
and	O
has	O
a	O
larger	O
number	O
of	O
possible	O
output	O
classes	O
.	O
We	O
assume	O
that	O
in	O
such	O
a	O
setting	O
,	O
adding	O
more	O
abstraction	O
power	O
to	O
the	O
model	O
(	O
in	O
the	O
form	O
of	O
a	O
BiLSTM	O
)	O
leads	O
to	O
better	O
results	O
.	O
For	O
a	O
more	O
detailed	O
analysis	O
,	O
Table	O
8	O
shows	O
the	O
slot	O
-	O
wise	O
results	O
for	O
the	O
non	O
-	O
neural	O
CRF	O
baseline	O
and	O
the	O
model	O
that	O
performs	O
best	O
on	O
the	O
development	O
set	O
:	O
BiLSTM	O
with	O
SciBERT	O
embeddings	O
.	O
As	O
in	O
the	O
case	O
of	O
entity	O
mention	O
detection	O
,	O
the	O
models	O
do	O
well	O
for	O
the	O
categories	O
that	O
consist	O
of	O
numeric	O
mentions	O
plus	O
particular	O
units	O
.	O
In	O
general	O
,	O
model	O
performance	O
is	O
also	O
tied	O
to	O
the	O
frequency	O
of	O
the	O
slot	O
types	O
in	O
the	O
dataset	O
.	O
Recognizing	O
the	O
role	O
a	O
material	O
plays	O
in	O
an	O
experiment	O
(	O
e.g.	O
,	O
AnodeMaterial	O
vs.	O
CathodeMaterial	O
)	O
remains	O
challenging	O
,	O
possibly	O
requiring	O
background	O
domain	O
knowledge	O
.	O
This	O
type	O
of	O
information	O
is	O
often	O
not	O
stated	O
explicitly	O
in	O
the	O
sentence	O
,	O
but	O
introduced	O
earlier	O
in	O
the	O
discourse	O
and	O
would	O
hence	O
require	O
document	O
-	O
level	O
modeling	O
.	O

Entity	O
Extraction	O
Evaluation	O
on	O
the	O
Synthesis	O
Procedures	O
Dataset	O

As	O
described	O
in	O
Section	O
2	O
,	O
the	O
data	O
set	O
curated	O
by	O
Mysore	O
et	O
al	O
.	O
(	O
2019	O
)	O
contains	O
230	O
synthesis	O
procedures	O
annotated	O
with	O
entity	O
type	O
information	O
.	O
11	O
We	O
apply	O
our	O
models	O
to	O
this	O
entity	O
extraction	O
task	O
in	O
order	O
to	O
estimate	O
the	O
degree	O
of	O
transferability	O
of	O
our	O
findings	O
to	O
similar	O
data	O
sets	O
.	O
To	O
the	O
best	O
of	O
11	O
our	O
knowledge	O
,	O
there	O
have	O
not	O
yet	O
been	O
any	O
publications	O
on	O
the	O
automatic	O
modeling	O
of	O
this	O
data	O
set	O
.	O
We	O
hence	O
compare	O
to	O
the	O
previous	O
work	O
of	O
Mysore	O
et	O
al	O
.	O
(	O
2017	O
)	O
,	O
who	O
perform	O
action	O
graph	O
induction	O
on	O
a	O
similar	O
data	O
set	O
.	O
12	O
Our	O
implementation	O
of	O
BiLSTM	O
-	O
CRF	O
mat2vec+word2vec	O
roughly	O
corresponds	O
to	O
their	O
BiLSTM	O
-	O
CRF	O
system	O
.	O

Table	O
9	O
shows	O
the	O
performance	O
of	O
our	O
models	O
when	O
trained	O
and	O
evaluated	O
on	O
the	O
synthesis	O
procedures	O
dataset	O
.	O
Detailed	O
scores	O
by	O
entity	O
type	O
can	O
be	O
found	O
in	O
the	O
Supplementary	O
Material	O
.	O
We	O
chose	O
to	O
use	O
the	O
data	O
split	O
suggested	O
by	O
the	O
authors	O
for	O
the	O
NER	O
task	O
,	O
using	O
200	O
documents	O
for	O
training	O
,	O
and	O
15	O
documents	O
for	O
each	O
dev	O
and	O
test	O
set	O
.	O
Among	O
the	O
non	O
-	O
BERT	O
-	O
based	O
systems	O
,	O
the	O
BiLSTM	O
variant	O
using	O
both	O
mat2vec	O
and	O
word2vec	O
performs	O
best	O
,	O
indicating	O
that	O
the	O
two	O
pre	O
-	O
trained	O
embeddings	O
contain	O
complementary	O
information	O
with	O
regard	O
to	O
this	O
task	O
.	O
The	O
best	O
performance	O
is	O
reached	O
by	O
the	O
BiL	O
-	O
STM	O
model	O
including	O
word2vec	O
,	O
mat2vec	O
,	O
bpe	O
and	O
SciBERT	O
embeddings	O
,	O
with	O
92.2	O
micro	O
-	O
average	O
F1	O
providing	O
a	O
strong	O
baseline	O
for	O
future	O
work	O
.	O

We	O
have	O
presented	O
a	O
new	O
dataset	O
for	O
information	O
extraction	O
in	O
the	O
materials	O
science	O
domain	O
consisting	O
of	O
45	O
open	O
-	O
access	O
scientific	O
articles	O
related	O
to	O
solid	O
oxide	O
fuel	O
cells	O
.	O
Our	O
detailed	O
corpus	O
and	O
interannotator	O
agreement	O
studies	O
highlight	O
the	O
complexity	O
of	O
the	O
task	O
and	O
verify	O
the	O
high	O
annotation	O
quality	O
.	O
Based	O
on	O
the	O
annotated	O
structures	O
,	O
we	O
suggest	O
three	O
information	O
extraction	O
tasks	O
:	O
the	O
detection	O
of	O
experiment	O
-	O
describing	O
sentences	O
,	O
entity	O
mention	O
recognition	O
and	O
typing	O
,	O
and	O
experiment	O
slot	O
filling	O
.	O
We	O
have	O
presented	O
various	O
strong	O
baselines	O
for	O
them	O
,	O
generally	O
finding	O
that	O
BERT	O
-	O
based	O
models	O
outperform	O
other	O
model	O
variants	O
.	O
While	O
some	O
categories	O
remain	O
challenging	O
,	O
overall	O
,	O
our	O
models	O
show	O
solid	O
performance	O
and	O
thus	O
prove	O
that	O
this	O
type	O
of	O
data	O
modeling	O
is	O
feasible	O
and	O
can	O
lead	O
to	O
systems	O
that	O
are	O
applicable	O
in	O
production	O
settings	O
.	O
Along	O
with	O
this	O
paper	O
,	O
we	O
make	O
the	O
annotation	O
guidelines	O
and	O
the	O
annotated	O
data	O
freely	O
available	O
.	O

Outlook	O
.	O
In	O
Section	O
7.1	O
,	O
we	O
have	O
shown	O
that	O
our	O
findings	O
generalize	O
well	O
by	O
applying	O
model	O
architectures	O
developed	O
on	O
our	O
corpus	O
to	O
another	O
dataset	O
.	O
A	O
natural	O
next	O
step	O
is	O
to	O
combine	O
the	O
datasets	O
in	O
a	O
multi	O
-	O
task	O
setting	O
to	O
investigate	O
to	O
what	O
extent	O
models	O
can	O
profit	O
from	O
combining	O
the	O
information	O
annotated	O
in	O
the	O
respective	O
datasets	O
.	O
Further	O
research	O
will	O
investigate	O
the	O
joint	O
modeling	O
of	O
entity	O
extraction	O
,	O
typing	O
and	O
experiment	O
frame	O
recognition	O
.	O
In	O
addition	O
,	O
there	O
are	O
also	O
further	O
natural	O
language	O
processing	O
tasks	O
that	O
can	O
be	O
researched	O
using	O
our	O
dataset	O
.	O
They	O
include	O
the	O
detection	O
of	O
events	O
and	O
sub	O
-	O
events	O
when	O
regarding	O
the	O
experiment	O
-	O
descriptions	O
as	O
events	O
,	O
and	O
a	O
more	O
linguistically	O
motivated	O
evaluation	O
of	O
the	O
framesemantic	O
approach	O
to	O
experiment	O
descriptions	O
in	O
text	O
,	O
e.g.	O
,	O
moving	O
away	O
from	O
the	O
one	O
-	O
experimentper	O
-	O
sentence	O
and	O
one	O
-	O
sentence	O
-	O
per	O
-	O
experiment	O
assumptions	O
and	O
modeling	O
the	O
graph	O
-	O
based	O
structures	O
as	O
annotated	O
.	O

Table	O
12	O
reports	O
full	O
statistics	O
for	O
the	O
task	O
of	O
identifying	O
experiment	O
-	O
describing	O
sentences	O
,	O
including	O
precision	O
and	O
recall	O
in	O
the	O
dev	O
setting	O
.	O

Position	O
encoding	O
(	O
PE	O
)	O
,	O
an	O
essential	O
part	O
of	O
self	O
-	O
attention	O
networks	O
(	O
SANs	O
)	O
,	O
is	O
used	O
to	O
preserve	O
the	O
word	O
order	O
information	O
for	O
natural	O
language	O
processing	O
tasks	O
,	O
generating	O
fixed	O
position	O
indices	O
for	O
input	O
sequences	O
.	O
However	O
,	O
in	O
cross	O
-	O
lingual	O
scenarios	O
,	O
e.g.	O
,	O
machine	O
translation	O
,	O
the	O
PEs	O
of	O
source	O
and	O
target	O
sentences	O
are	O
modeled	O
independently	O
.	O
Due	O
to	O
word	O
order	O
divergences	O
in	O
different	O
languages	O
,	O
modeling	O
the	O
cross	O
-	O
lingual	O
positional	O
relationships	O
might	O
help	O
SANs	O
tackle	O
this	O
problem	O
.	O
In	O
this	O
paper	O
,	O
we	O
augment	O
SANs	O
with	O
crosslingual	O
position	O
representations	O
to	O
model	O
the	O
bilingually	O
aware	O
latent	O
structure	O
for	O
the	O
input	O
sentence	O
.	O
Specifically	O
,	O
we	O
utilize	O
bracketing	O
transduction	O
grammar	O
(	O
BTG)-based	O
reordering	O
information	O
to	O
encourage	O
SANs	O
to	O
learn	O
bilingual	O
diagonal	O
alignments	O
.	O
Experimental	O
results	O
on	O
WMT'14	O
English⇒German	O
,	O
WAT'17	O
Japanese⇒English	O
,	O
and	O
WMT'17	O
Chinese⇔English	O
translation	O
tasks	O
demonstrate	O
that	O
our	O
approach	O
significantly	O
and	O
consistently	O
improves	O
translation	O
quality	O
over	O
strong	O
baselines	O
.	O
Extensive	O
analyses	O
confirm	O
that	O
the	O
performance	O
gains	O
come	O
from	O
the	O
cross	O
-	O
lingual	O
information	O
.	O

Although	O
self	O
-	O
attention	O
networks	O
(	O
SANs	O
)	O
(	O
Lin	O
et	O
al	O
.	O
,	O
2017	O
)	O
have	O
achieved	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
performance	O
on	O
several	O
natural	O
language	O
processing	O
(	O
NLP	O
)	O
tasks	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017;Devlin	O
et	O
al	O
.	O
,	O
2019;Radford	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
they	O
possess	O
the	O
innate	O
disadvantage	O
of	O
sequential	O
modeling	O
due	O
to	O
the	O
lack	O
of	O
positional	O
information	O
.	O
Therefore	O
,	O
absolute	O
position	O
encoding	O
(	O
APE	O
)	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
and	O
relative	O
position	O
encoding	O
(	O
RPE	O
)	O
(	O
Shaw	O
et	O
al	O
.	O
,	O
2018	O
)	O
were	O
introduced	O
to	O
better	O
capture	O
the	O
sequential	O
dependencies	O
.	O
However	O
,	O
either	O
absolute	O
or	O
relative	O
PE	O
is	O
language	O
-	O
independent	O
and	O
its	O
embedding	O
remains	O
fixed	O
.	O
This	O
inhibits	O
the	O
capacity	O
of	O
SANs	O
when	O
modelling	O
multiple	O
languages	O
,	O
which	O
have	O
diverse	O
word	O
orders	O
and	O
structures	O
(	O
Gell	O
-	O
Mann	O
and	O
Ruhlen	O
,	O
2011	O
)	O
.	O
Recent	O
work	O
have	O
shown	O
that	O
modeling	O
cross	O
-	O
lingual	O
information	O
(	O
e.g.	O
,	O
alignment	O
or	O
reordering	O
)	O
at	O
encoder	O
or	O
attention	O
level	O
improves	O
translation	O
performance	O
for	O
different	O
language	O
pairs	O
(	O
Cohn	O
et	O
al	O
.	O
,	O
2016;Du	O
and	O
Way	O
,	O
2017;Zhao	O
et	O
al	O
.	O
,	O
2018;Kawara	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
Inspired	O
by	O
their	O
work	O
,	O
we	O
propose	O
to	O
augment	O
SANs	O
with	O
cross	O
-	O
lingual	O
representations	O
,	O
by	O
encoding	O
reordering	O
indices	O
at	O
embedding	O
level	O
.	O
Taking	O
English⇒Chinese	O
translation	O
task	O
for	O
example	O
,	O
we	O
first	O
reorder	O
the	O
English	O
sentence	O
by	O
deriving	O
a	O
latent	O
bracketing	O
transduction	O
grammar	O
(	O
BTG	O
)	O
tree	O
(	O
Wu	O
,	O
1997	O
)	O
(	O
Fig	O
.	O
1a	O
)	O
.	O
Similar	O
to	O
absolute	O
position	O
,	O
the	O
reordering	O
information	O
can	O
be	O
represented	O
as	O
cross	O
-	O
lingual	O
position	O
(	O
Fig	O
.	O
1b	O
)	O
.	O
In	O
addition	O
,	O
we	O
propose	O
two	O
strategies	O
to	O
incorporate	O
cross	O
-	O
lingual	O
position	O
encoding	O
into	O
SANs	O
.	O
We	O
conducted	O
experiments	O
on	O
three	O
commonlycited	O
datasets	O
of	O
machine	O
translation	O
.	O
Results	O
show	O
that	O
exploiting	O
cross	O
-	O
lingual	O
PE	O
consistently	O
improves	O
translation	O
quality	O
.	O
Further	O
analysis	O
reveals	O
that	O
our	O
method	O
improves	O
the	O
alignment	O
quality	O
(	O
§	O
Sec	O
.	O
4.3	O
)	O
and	O
context	O
-	O
free	O
Transformer	O
(	O
Tang	O
et	O
al	O
.	O
,	O
2019	O
)	O
(	O
§	O
Sec	O
.	O
4.4	O
)	O
.	O
Furthermore	O
,	O
contrastive	O
evaluation	O
demonstrates	O
that	O
NMT	O
models	O
benefits	O
from	O
the	O
cross	O
-	O
lingual	O
information	O
rather	O
than	O
denoising	O
ability	O
(	O
§	O
Sec	O
.	O
4.5	O
)	O
.	O

Position	O
Encoding	O
To	O
tackle	O
the	O
position	O
unaware	O
problem	O
,	O
absolute	O
position	O
information	O
is	O
injected	O
into	O
the	O
SANs	O
:	O

Self	O
-	O
Attention	O
The	O
SANs	O
compute	O
the	O
attention	O
of	O
each	O
pair	O
of	O
elements	O
in	O
parallel	O
.	O
It	O
first	O
converts	O
the	O
input	O
into	O
three	O
matrices	O
Q	O
,	O
K	O
,	O
V	O
,	O
representing	O
queries	O
,	O
keys	O
,	O
and	O
values	O
,	O
respectively	O
:	O

SANs	O
can	O
be	O
implemented	O
with	O
multi	O
-	O
head	O
attention	O
mechanism	O
,	O
which	O
requires	O
extra	O
splitting	O
and	O
concatenation	O
operations	O
.	O
Specifically	O
,	O
W	O
Q	O
,	O
W	O
K	O
,	O
W	O
V	O
and	O
Q	O
,	O
K	O
,	O
V	O
in	O
Eq	O
.	O
(	O
3	O
)	O
is	O
split	O
into	O
H	O
sub	O
-	O
matrices	O
,	O
yielding	O
H	O
heads	O
.	O
For	O
the	O
h	O
-	O
th	O
head	O
,	O
the	O
output	O
is	O
computed	O
by	O
:	O

First	O
,	O
we	O
built	O
a	O
BTG	O
-	O
based	O
reordering	O
model	O
(	O
Neubig	O
et	O
al	O
.	O
,	O
2012	O
)	O
to	O
generate	O
a	O
reordered	O
source	O
sentence	O
according	O
to	O
the	O
word	O
order	O
of	O
its	O
corresponding	O
target	O
sentence	O
.	O
Second	O
,	O
we	O
obtained	O
the	O
reordered	O
word	O
indices	O
pos	O
XL	O
that	O
correspond	O
with	O
the	O
input	O
sentence	O
X.	O
To	O
output	O
the	O
cross	O
-	O
lingual	O
position	O
matrix	O
PE	O
XL	O
,	O
we	O
inherit	O
the	O
sinusoidal	O
function	O
in	O
Eq	O
.	O
(	O
1	O
)	O
.	O
Formally	O
,	O
the	O
process	O
is	O
:	O

As	O
shown	O
in	O
Fig	O
.	O
2	O
,	O
we	O
propose	O
two	O
strategies	O
to	O
integrate	O
the	O
cross	O
-	O
lingual	O
position	O
encoding	O
(	O
XL	O
PE	O
)	O
into	O
SANs	O
:	O
inputting	O
-	O
level	O
XL	O
(	O
InXL	O
)	O
SANs	O
and	O
head	O
-	O
level	O
(	O
HeadXL	O
)	O
SANs	O
.	O

Inputting	O
-	O
level	O
XL	O
SANs	O
As	O
illustrated	O
in	O
Fig	O
.	O
2a	O
,	O
we	O
employ	O
a	O
non	O
-	O
linear	O
function	O
TANH(•	O
)	O
to	O
fuse	O
PE	O
abs	O
and	O
PE	O
XL	O
:	O

Similarly	O
,	O
we	O
use	O
Eq	O
.	O
(	O
3)∼	O
(	O
5	O
)	O
to	O
calculate	O
multiple	O
heads	O
of	O
SANs	O
.	O

Head	O
-	O
level	O
XL	O
SANs	O
Instead	O
of	O
projecting	O
XL	O
PE	O
to	O
all	O
attention	O
heads	O
,	O
we	O
feed	O
partial	O
of	O
them	O
,	O
such	O
that	O
some	O
heads	O
contain	O
XL	O
PE	O
and	O
others	O
contain	O
APE	O
,	O
namely	O
HeadXL	O
.	O
As	O
shown	O
in	O
Fig	O
.	O
2b	O
,	O
we	O
fist	O
add	O
APE	O
and	O
XL	O
PE	O
for	O
X	O
,	O
respectively	O
:	O

We	O
denote	O
the	O
number	O
of	O
XL	O
PE	O
equipped	O
heads	O
as	O
τ	O
∈	O
{	O
0	O
,	O
.	O
.	O
.	O
,	O
H	O
}	O
.	O
To	O
perform	O
the	O
attention	O
calculation	O
,	O

In	O
particular	O
,	O
τ	O
=	O
0	O
refers	O
to	O
the	O
original	O
Transformer	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
and	O
τ	O
=	O
H	O
means	O
that	O
XL	O
PE	O
will	O
propagate	O
over	O
all	O
attention	O
heads	O
.	O

We	O
conduct	O
experiments	O
on	O
word	O
order	O
-	O
diverse	O
language	O
pairs	O
:	O
WMT'14	O
English⇒German	O
(	O
En	O
-	O
De	O
)	O
,	O
WAT'17	O
Japanese⇒English	O
(	O
Ja	O
-	O
En	O
)	O
,	O
and	O
WMT'17	O
Chinese⇔English	O
(	O
Zh	O
-	O
En	O
&	O
En	O
-	O
Zh	O
)	O
.	O

For	O
English⇒German	O
,	O
the	O
training	O
set	O
consists	O
of	O
4.5	O
million	O
sentence	O
pairs	O
and	O
newstest2013	O
&	O
2014	O
are	O
used	O
as	O
the	O
dev	O
.	O
and	O
test	O
sets	O
,	O
respectively	O
.	O
BPE	O
with	O
32	O
K	O
merge	O
operations	O
is	O
used	O
to	O
handle	O
low	O
-	O
frequency	O
words	O
.	O
For	O
Japanese⇒English	O
,	O
we	O
follow	O
Morishita	O
et	O
al	O
.	O
(	O
2017	O
)	O
to	O
use	O
the	O
first	O
two	O
sections	O
as	O
training	O
data	O
,	O
which	O
consists	O
of	O
2.0	O
million	O
sentence	O
pairs	O
.	O
The	O
dev	O
.	O
and	O
test	O
sets	O
contain	O
1790	O
and	O
1812	O
sentences	O
.	O
For	O
Chinese⇔English	O
,	O
we	O
follow	O
Hassan	O
et	O
al	O
.	O
(	O
2018	O
)	O
sentence	O
pairs	O
.	O
We	O
develop	O
on	O
devtest2017	O
and	O
test	O
on	O
newstest2017	O
.	O
We	O
use	O
SacreBLEU	O
(	O
Post	O
,	O
2018	O
)	O
as	O
the	O
evaluation	O
metric	O
with	O
statistical	O
significance	O
test	O
(	O
Collins	O
et	O
al	O
.	O
,	O
2005	O
)	O
.	O
We	O
evaluate	O
the	O
proposed	O
XL	O
PE	O
strategies	O
on	O
Transformer	O
.	O
The	O
baseline	O
systems	O
include	O
Relative	O
PE	O
(	O
Shaw	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
directional	O
SAN	O
(	O
DiSAN	O
,	O
Shen	O
et	O
al	O
.	O
2018	O
)	O
.	O
We	O
implement	O
them	O
on	O
top	O
of	O
OpenNMT	O
(	O
Klein	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
In	O
addition	O
,	O
we	O
report	O
the	O
results	O
of	O
previous	O
studies	O
(	O
Hao	O
et	O
al	O
.	O
,	O
2019;Chen	O
et	O
al	O
.	O
,	O
2019b	O
,	O
a;Du	O
and	O
Way	O
,	O
2017;Hassan	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O

The	O
reordered	O
source	O
sentences	O
are	O
generated	O
by	O
BTG	O
-	O
based	O
preordering	O
model	O
(	O
Neubig	O
et	O
al	O
.	O
,	O
2012	O
)	O
trained	O
with	O
above	O
sub	O
-	O
word	O
level	O
1	O
parallel	O
corpus	O
.	O
At	O
training	O
phase	O
,	O
we	O
first	O
obtain	O
word	O
alignments	O
from	O
parallel	O
data	O
using	O
GIZA++	O
or	O
FastAlign	O
,	O
and	O
then	O
the	O
training	O
process	O
is	O
to	O
find	O
the	O
optimal	O
BTG	O
tree	O
for	O
source	O
sentence	O
consistent	O
with	O
the	O
order	O
of	O
the	O
target	O
sentence	O
based	O
on	O
the	O
word	O
alignments	O
and	O
parallel	O
data	O
.	O
At	O
decoding	O
phase	O
,	O
we	O
only	O
provide	O
source	O
sentences	O
as	O
input	O
and	O
the	O
model	O
can	O
output	O
reordering	O
indices	O
,	O
which	O
will	O
be	O
fed	O
into	O
NMT	O
model	O
.	O
Thus	O
,	O
bilingual	O
alignment	O
information	O
is	O
only	O
used	O
to	O
preprocess	O
training	O
data	O
,	O
but	O
not	O
necessary	O
at	O
decoding	O
time	O
.	O

For	O
fair	O
comparison	O
,	O
we	O
keep	O
the	O
Transformer	O
decoder	O
unchanged	O
and	O
validate	O
different	O
position	O
representation	O
strategies	O
on	O
the	O
encoder	O
.	O
We	O
conduct	O
all	O
experiments	O
on	O
the	O
TRANSFORMER	O
-	O
BIG	O
with	O
four	O
V100	O
GPUs	O
.	O

Effect	O
of	O
τ	O
in	O
HeadXL	O
SANs	O

Fig	O
.	O
3	O
reports	O
the	O
results	O
of	O
different	O
τ	O
for	O
Head	O
XL	O
SANs	O
.	O
With	O
increasing	O
of	O
XL	O
PE	O
-	O
informed	O
heads	O
,	O
the	O
best	O
BLEU	O
is	O
achieved	O
when	O
#	O
heads	O
=	O
4	O
,	O
which	O
is	O
therefore	O
left	O
as	O
the	O
default	O
setting	O
for	O
HeadXL	O
.	O
Then	O
,	O
the	O
BLEU	O
score	O
gradually	O
decreases	O
as	O
the	O
#	O
System	O
Architecture	O
BLEU	O
#	O
Param	O
.	O
number	O
of	O
APE	O
-	O
informed	O
heads	O
decrease	O
(	O
τ	O
↑	O
)	O
,	O
indicating	O
that	O
sequential	O
position	O
embedding	O
is	O
still	O
essential	O
for	O
SANs	O
.	O

Tab	O
.	O
1	O
shows	O
the	O
results	O
on	O
En	O
-	O
De	O
,	O
inputting	O
-	O
level	O
cross	O
-	O
lingual	O
PE	O
(	O
+	O
InXL	O
PE	O
)	O
and	O
head	O
-	O
level	O
crosslingual	O
PE	O
(	O
+	O
HeadXL	O
PE	O
)	O
outperform	O
Transformer	O
BIG	O
by	O
0.30	O
and	O
0.36	O
BLEU	O
points	O
,	O
and	O
combining	O
these	O
two	O
strategies	O
2	O
achieves	O
a	O
0.69	O
BLEU	O
point	O
increase	O
.	O
For	O
Ja	O
-	O
En	O
,	O
Zh	O
-	O
En	O
,	O
and	O
En	O
-	O
Zh	O
(	O
Tab	O
.	O
2	O
)	O
,	O
we	O
observe	O
a	O
similar	O
phenomenon	O
,	O
demonstrating	O
that	O
XL	O
PE	O
on	O
SANs	O
do	O
improve	O
the	O
translation	O
performance	O
for	O
several	O
language	O
pairs	O
.	O
It	O
is	O
worth	O
noting	O
that	O
our	O
approach	O
introduces	O
nearly	O
no	O
additional	O
parameters	O
(	O
+0.01	O
M	O
over	O
282.55	O
M	O
)	O
.	O

Our	O
proposed	O
XL	O
PE	O
intuitively	O
encourages	O
SANs	O
to	O
learn	O
bilingual	O
diagonal	O
alignment	O
,	O
so	O
has	O
the	O
2	O
Replace	O
PEXL	O
in	O
Eq	O
.	O
(	O
9	O
)	O
with	O
PEIN	O
-	O
XL	O
in	O
Eq	O
.	O
(	O
8)	O
.	O
potential	O
to	O
induce	O
better	O
attention	O
matrices	O
.	O
We	O
explore	O
this	O
hypothesis	O
on	O
the	O
widely	O
used	O
Gold	O
Alignment	O
dataset	O
3	O
and	O
follow	O
Tang	O
et	O
al	O
.	O
(	O
2019	O
)	O
to	O
perform	O
the	O
alignment	O
.	O
The	O
only	O
difference	O
being	O
that	O
we	O
average	O
the	O
attention	O
matrices	O
across	O
all	O
heads	O
from	O
the	O
penultimate	O
layer	O
(	O
Garg	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
The	O
alignment	O
error	O
rate	O
(	O
AER	O
,	O
Och	O
and	O
Ney	O
2003	O
)	O
,	O
precision	O
(	O
P	O
)	O
and	O
recall	O
(	O
R	O
)	O
are	O
reported	O
as	O
the	O
evaluation	O
metrics	O
.	O
Tab	O
.	O
3	O
summarizes	O
the	O
results	O
.	O
We	O
can	O
see	O
:	O
1	O
)	O
XL	O
PE	O
allows	O
SANs	O
to	O
learn	O
better	O
attention	O
matrices	O
,	O
thereby	O
improving	O
alignment	O
performance	O
(	O
27.4	O
/	O
26.9	O
vs.	O
29.7	O
)	O
;	O
and	O
2	O
)	O
combining	O
the	O
two	O
strategies	O
delivers	O
consistent	O
improvements	O
(	O
24.7	O
vs.	O
29.7	O
)	O
.	O

Augmenting	O
SANs	O
with	O
position	O
representation	O
SANs	O
ignore	O
the	O
position	O
of	O
each	O
token	O
due	O
to	O
its	O
position	O
-	O
unaware	O
"	O
bag	O
-	O
of	O
-	O
words	O
"	O
assumption	O
.	O
The	O
most	O
straightforward	O
strategy	O
is	O
adding	O
the	O
position	O
representations	O
as	O
part	O
of	O
the	O
token	O
representations	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017;Shaw	O
et	O
al	O
.	O
,	O
2018	O
lingual	O
position	O
information	O
between	O
languages	O
.	O

Modeling	O
cross	O
-	O
lingual	O
divergence	O
There	O
has	O
been	O
many	O
works	O
modeling	O
cross	O
-	O
lingual	O
divergence	O
(	O
e.g.	O
,	O
reordering	O
)	O
in	O
statistical	O
machine	O
translation	O
(	O
Nagata	O
et	O
al	O
.	O
,	O
2006;Durrani	O
et	O
al	O
.	O
,	O
2011Durrani	O
et	O
al	O
.	O
,	O
,	O
2013	O
.	O
However	O
,	O
it	O
is	O
difficult	O
to	O
migrant	O
them	O
to	O
neural	O
machine	O
translation	O
.	O
Kawara	O
et	O
al	O
.	O
(	O
2018	O
)	O
pre	O
-	O
reordered	O
the	O
source	O
sentences	O
with	O
a	O
recursive	O
neural	O
network	O
model	O
.	O
Chen	O
et	O
al	O
.	O
(	O
2019a	O
)	O
learned	O
the	O
reordering	O
embedding	O
by	O
considering	O
the	O
relationship	O
between	O
the	O
position	O
embedding	O
of	O
a	O
word	O
and	O
SANS	O
-	O
calculated	O
sentence	O
representation	O
.	O
showed	O
that	O
SANs	O
in	O
machine	O
translation	O
could	O
learn	O
word	O
order	O
mainly	O
due	O
to	O
the	O
PE	O
,	O
indicating	O
that	O
modeling	O
cross	O
-	O
lingual	O
information	O
at	O
position	O
representation	O
level	O
may	O
be	O
informative	O
.	O
Thus	O
,	O
we	O
propose	O
a	O
novel	O
cross	O
-	O
lingual	O
PE	O
method	O
to	O
improve	O
SANs	O
.	O

In	O
this	O
paper	O
,	O
we	O
presented	O
a	O
novel	O
cross	O
-	O
lingual	O
position	O
encoding	O
to	O
augment	O
SANs	O
by	O
considering	O
cross	O
-	O
lingual	O
information	O
(	O
i.e.	O
,	O
reordering	O
indices	O
)	O
for	O
the	O
input	O
sentence	O
.	O
We	O
designed	O
two	O
strategies	O
to	O
integrate	O
it	O
into	O
SANs	O
.	O
Experiments	O
indicated	O
that	O
the	O
proposed	O
strategies	O
consistently	O
improve	O
the	O
translation	O
performance	O
.	O
In	O
the	O
future	O
,	O
we	O
plan	O
to	O
extend	O
the	O
cross	O
-	O
lingual	O
position	O
encoding	O
to	O
non	O
-	O
autoregressive	O
MT	O
(	O
Gu	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
unsupervised	O
NMT	O
(	O
Lample	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O

We	O
harness	O
neural	O
language	O
and	O
commonsense	O
models	O
to	O
study	O
how	O
cognitive	O
processes	O
of	O
recollection	O
and	O
imagination	O
are	O
engaged	O
in	O
storytelling	O
.	O
We	O
rely	O
on	O
two	O
key	O
aspects	O
of	O
stories	O
:	O
narrative	O
flow	O
(	O
how	O
the	O
story	O
reads	O
)	O
and	O
semantic	O
vs.	O
episodic	O
knowledge	O
(	O
the	O
types	O
of	O
events	O
in	O
the	O
story	O
)	O
.	O
We	O
propose	O
as	O
a	O
measure	O
of	O
narrative	O
flow	O
the	O
likelihood	O
of	O
sentences	O
under	O
generative	O
language	O
models	O
conditioned	O
on	O
varying	O
amounts	O
of	O
history	O
.	O
Then	O
,	O
we	O
quantify	O
semantic	O
knowledge	O
by	O
measuring	O
the	O
frequency	O
of	O
commonsense	O
events	O
(	O
from	O
the	O
ATOMIC	O
knowledge	O
graph	O
;	O
Sap	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
and	O
episodic	O
knowledge	O
by	O
counting	O
realis	O
events	O
(	O
Sims	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
both	O
shown	O
in	O
Figure	O
1	O
.	O

We	O
introduce	O
HIPPOCORPUS	O
,	O
1	O
a	O
dataset	O
of	O
6,854	O
diary	O
-	O
like	O
short	O
stories	O
about	O
salient	O
life	O
events	O
,	O
to	O
examine	O
the	O
cognitive	O
processes	O
of	O
remembering	O
and	O
imagining	O
.	O
Using	O
a	O
crowdsourcing	O
pipeline	O
,	O
we	O
collect	O
pairs	O
of	O
recalled	O
and	O
imagined	O
stories	O
written	O
about	O
the	O
same	O
topic	O
.	O
By	O
design	O
,	O
authors	O
of	O
recalled	O
stories	O
rely	O
on	O
their	O
episodic	O
memory	O
to	O
tell	O
their	O
story	O
.	O

We	O
demonstrate	O
that	O
our	O
measures	O
can	O
uncover	O
differences	O
in	O
imagined	O
and	O
recalled	O
stories	O
in	O
HIPPOCORPUS	O
.	O
Imagined	O
stories	O
contain	O
more	O
commonsense	O
events	O
and	O
elaborations	O
,	O
whereas	O
recalled	O
stories	O
are	O
more	O
dense	O
in	O
concrete	O
events	O
.	O
Additionally	O
,	O
imagined	O
stories	O
flow	O
substantially	O
more	O
linearly	O
than	O
recalled	O
stories	O
.	O
Our	O
findings	O
provide	O
evidence	O
that	O
surface	O
language	O
reflects	O
the	O
differences	O
in	O
cognitive	O
processes	O
used	O
in	O
imagining	O
and	O
remembering	O
.	O

We	O
construct	O
HIPPOCORPUS	O
,	O
containing	O
6,854	O
stories	O
(	O
Table	O
1	O
)	O
,	O
to	O
enable	O
the	O
study	O
of	O
imagined	O
and	O
recalled	O
stories	O
,	O
as	O
most	O
prior	O
corpora	O
are	O
either	O
limited	O
in	O
size	O
or	O
topic	O
(	O
e.g.	O
,	O
Greenberg	O
et	O
al	O
.	O
,	O
1996;Ott	O
et	O
al	O
.	O
,	O
2011	O
)	O
.	O
See	O
Appendix	O
A	O
for	O
additional	O
details	O
(	O
e.g.	O
,	O
worker	O
demographics	O
;	O
§	O
A.2	O
)	O
.	O

Inspired	O
by	O
recent	O
work	O
on	O
discourse	O
modeling	O
(	O
Kang	O
et	O
al	O
.	O
,	O
2019;Nadeem	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
we	O
use	O
language	O
models	O
to	O
assess	O
the	O
narrative	O
linearity	O
of	O
a	O
story	O
by	O
measuring	O
how	O
sentences	O
relate	O
to	O
their	O
context	O
in	O
the	O
story	O
.	O
We	O
compare	O
the	O
likelihoods	O
of	O
sentences	O
under	O
two	O
generative	O
models	O
(	O
Figure	O
2	O
)	O
.	O
The	O
bag	O
model	O
makes	O
the	O
assumption	O
that	O
every	O
sentence	O
is	O
drawn	O
independently	O
from	O
the	O
main	O
theme	O
of	O
the	O
story	O
(	O
represented	O
by	O
E	O
)	O
.	O
On	O
the	O
other	O
hand	O
,	O
the	O
chain	O
model	O
assumes	O
that	O
a	O
story	O
begins	O
with	O
a	O
theme	O
,	O
and	O
sentences	O
linearly	O
follow	O
each	O
other	O
.	O
3	O
.	O
∆	O
l	O
is	O
computed	O
as	O
the	O
difference	O
in	O
negative	O
loglikelihoods	O
between	O
the	O
bag	O
and	O
chain	O
models	O
:	O

We	O
train	O
a	O
realis	O
event	O
tagger	O
(	O
using	O
BERT	O
-	O
base	O
;	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
on	O
the	O
annotated	O
literary	O
events	O
corpus	O
by	O
Sims	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
which	O
slightly	O
outperforms	O
the	O
original	O
author	O
's	O
models	O
.	O
We	O
provide	O
further	O
training	O
details	O
in	O
Appendix	O
B.1	O
.	O

Given	O
the	O
social	O
focus	O
of	O
our	O
stories	O
,	O
we	O
use	O
the	O
social	O
commonsense	O
knowledge	O
graph	O
ATOMIC	O
(	O
Sap	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
4	O
For	O
each	O
story	O
,	O
we	O
first	O
match	O
possible	O
ATOMIC	O
events	O
to	O
sentences	O
by	O
selecting	O
events	O
that	O
share	O
noun	O
chunks	O
and	O
verb	O
phrases	O
with	O
sentences	O
(	O
e.g.	O
,	O
"	O
getting	O
married	O
"	O
"	O
Per	O
-	O
sonX	O
gets	O
married	O
"	O
;	O
Figure	O
1	O
)	O
.	O
We	O
then	O
search	O
the	O
matched	O
sentences	O
'	O
surrounding	O
sentences	O
for	O
commonsense	O
inferences	O
(	O
e.g.	O
,	O
"	O
be	O
very	O
happy	O
"	O
"	O
happy	O
"	O
;	O
Figure	O
1	O
)	O
.	O
We	O
describe	O
this	O
algorithm	O
in	O
further	O
detail	O
in	O
Appendix	O
B.2	O
.	O
In	O
our	O
analyses	O
,	O
the	O
measure	O
quantifies	O
the	O
number	O
of	O
story	O
sentences	O
with	O
commonsense	O
tuple	O
matches	O
in	O
the	O
two	O
preceding	O
and	O
following	O
sentences	O
.	O

To	O
supplement	O
our	O
analyses	O
,	O
we	O
compute	O
several	O
coarse	O
-	O
grained	O
lexical	O
counts	O
for	O
each	O
story	O
in	O
HIPPOCORPUS	O
.	O
Such	O
approaches	O
have	O
been	O
used	O
in	O
prior	O
efforts	O
to	O
investigate	O
author	O
mental	O
states	O
,	O
temporal	O
orientation	O
,	O
or	O
counterfactual	O
thinking	O
in	O
language	O
(	O
Tausczik	O
and	O
Pennebaker	O
,	O
2010;Schwartz	O
et	O
al	O
.	O
,	O
2015;Son	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O

We	O
count	O
psychologically	O
relevant	O
word	O
categories	O
using	O
the	O
Linguistic	O
Inquiry	O
Word	O
Count	O
(	O
Pennebaker	O
et	O
al	O
.	O
,	O
2015	O
,	O
LIWC	O
;)	O
,	O
focusing	O
only	O
on	O
the	O
cognitive	O
processes	O
,	O
positive	O
emotion	O
,	O
negative	O
emotion	O
,	O
and	O
I	O
-	O
word	O
categories	O
,	O
as	O
well	O
as	O
the	O
ANALYTIC	O
and	O
TONE	O
summary	O
variables	O
.	O
5	O
Additionally	O
,	O
we	O
measure	O
the	O
average	O
concreteness	O
level	O
of	O
words	O
in	O
stories	O
using	O
the	O
lexicon	O
by	O
Brysbaert	O
et	O
al	O
.	O
(	O
2014	O
)	O
.	O

We	O
summarize	O
the	O
differences	O
between	O
imagined	O
and	O
recalled	O
stories	O
in	O
HIPPOCORPUS	O
in	O
Table	O
2	O
.	O
For	O
our	O
narrative	O
flow	O
and	O
lexicon	O
-	O
based	O
analyses	O
,	O
4	O
ATOMIC	O
contains	O
social	O
and	O
inferential	O
knowledge	O
about	O
the	O
causes	O
(	O
e.g.	O
,	O
"	O
X	O
wants	O
to	O
start	O
a	O
family	O
"	O
)	O
and	O
effects	O
(	O
e.g.	O
,	O
"	O
X	O
throws	O
a	O
party	O
"	O
,	O
"	O
X	O
feels	O
loved	O
"	O
)	O
of	O
everyday	O
situations	O
like	O
"	O
PersonX	O
decides	O
to	O
get	O
married	O
"	O
.	O
5	O
See	O
liwc.wpengine.com/interpretingliwc-output/	O
for	O
more	O
information	O
on	O
LIWC	O
variables	O
.	O
we	O
perform	O
paired	O
t	O
-	O
tests	O
.	O
For	O
realis	O
and	O
commonsense	O
event	O
measures	O
,	O
we	O
perform	O
linear	O
regressions	O
controlling	O
for	O
story	O
length	O
.	O
6	O
We	O
Holmcorrect	O
for	O
multiple	O
comparisons	O
for	O
all	O
our	O
analyses	O
(	O
Holm	O
,	O
1979	O
)	O
.	O

First	O
,	O
we	O
compare	O
the	O
effects	O
of	O
recency	O
of	O
the	O
event	O
described	O
(	O
TIMESINCEEVENT	O
:	O
a	O
continuous	O
variable	O
representing	O
the	O
log	O
time	O
since	O
the	O
event	O
)	O
.	O
9	O
Then	O
,	O
we	O
contrast	O
recalled	O
stories	O
to	O
their	O
retold	O
counterparts	O
in	O
pairwise	O
comparisons	O
.	O
Finally	O
,	O
we	O
measure	O
the	O
effect	O
of	O
how	O
frequently	O
the	O
experienced	O
event	O
is	O
thought	O
or	O
talked	O
about	O
(	O
FREQUENCYOFRECALL	O
:	O
a	O
continuous	O
variable	O
ranging	O
from	O
very	O
rarely	O
to	O
very	O
frequently	O
)	O
.	O
10	O
As	O
in	O
§	O
4	O
,	O
we	O
Holm	O
-	O
correct	O
for	O
multiple	O
comparisons	O
.	O

Frequency	O
of	O
recall	O
.	O
We	O
find	O
that	O
the	O
more	O
an	O
event	O
is	O
thought	O
or	O
talked	O
about	O
(	O
i.e.	O
,	O
higher	O
FRE	O
-	O
QUENCYOFRECALL	O
)	O
,	O
the	O
more	O
linearly	O
its	O
story	O
flows	O
(	O
∆	O
l	O
;	O
|β|	O
=	O
0.07	O
,	O
p	O
<	O
0.001	O
)	O
,	O
and	O
the	O
fewer	O
realis	O
events	O
(	O
|β|	O
=	O
0.09	O
,	O
p	O
<	O
0.001	O
)	O
it	O
contains	O
.	O

Furthermore	O
,	O
using	O
lexicon	O
-	O
based	O
measures	O
,	O
we	O
find	O
that	O
stories	O
with	O
high	O
FREQUENCYOFRE	O
-	O
CALL	O
tend	O
to	O
contain	O
more	O
self	O
references	O
(	O
Iwords	O
;	O
Pearson	O
's	O
|r|	O
=	O
0.07	O
,	O
p	O
<	O
0.001	O
)	O
.	O
Conversely	O
,	O
stories	O
that	O
are	O
less	O
frequently	O
recalled	O
are	O
more	O
logical	O
or	O
hierarchical	O
(	O
LIWC	O
's	O
ANALYTIC	O
;	O
Pearson	O
's	O
|r|	O
=	O
0.09	O
,	O
p	O
<	O
0.001	O
)	O
and	O
more	O
concrete	O
(	O
Pearson	O
's	O
|r|	O
=	O
0.05	O
,	O
p	O
=	O
0.03	O
)	O
.	O

To	O
investigate	O
the	O
use	O
of	O
NLP	O
tools	O
for	O
studying	O
the	O
cognitive	O
traces	O
of	O
recollection	O
versus	O
imagination	O
in	O
stories	O
,	O
we	O
collect	O
and	O
release	O
HIP	O
-	O
POCORPUS	O
,	O
a	O
dataset	O
of	O
imagined	O
and	O
recalled	O
stories	O
.	O
We	O
introduce	O
measures	O
to	O
characterize	O
narrative	O
flow	O
and	O
influence	O
of	O
semantic	O
vs.	O
episodic	O
knowledge	O
in	O
stories	O
.	O
We	O
show	O
that	O
imagined	O
stories	O
have	O
a	O
more	O
linear	O
flow	O
and	O
contain	O
more	O
commonsense	O
knowledge	O
,	O
whereas	O
recalled	O
stories	O
are	O
less	O
connected	O
and	O
contain	O
more	O
specific	O
concrete	O
events	O
.	O
Additionally	O
,	O
we	O
show	O
that	O
our	O
measures	O
can	O
uncover	O
the	O
effect	O
in	O
language	O
of	O
narrativization	O
of	O
memories	O
over	O
time	O
.	O
We	O
hope	O
these	O
findings	O
bring	O
attention	O
to	O
the	O
feasibility	O
of	O
employing	O
statistical	O
natural	O
language	O
processing	O
machinery	O
as	O
tools	O
for	O
exploring	O
human	O
cognition	O
.	O
Figure	O
4	O
:	O
We	O
extract	O
phrases	O
from	O
the	O
main	O
themes	O
of	O
recalled	O
(	O
left	O
)	O
and	O
imagined	O
(	O
right	O
)	O
stories	O
,	O
using	O
RAKE	O
(	O
Rose	O
et	O
al	O
.	O
,	O
2010	O
)	O
;	O
size	O
of	O
words	O
corresponds	O
to	O
frequency	O
in	O
corpus	O
,	O
and	O
color	O
is	O
only	O
for	O
readability	O
.	O

To	O
detect	O
realis	O
events	O
in	O
our	O
stories	O
,	O
we	O
train	O
a	O
tagger	O
(	O
using	O
BERT	O
-	O
base	O
;	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
on	O
the	O
annotated	O
corpus	O
by	O
Sims	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O
This	O
corpus	O
contains	O
8k	O
realis	O
events	O
annotated	O
by	O
experts	O
in	O
sentences	O
drawn	O
from	O
100	O
English	O
books	O
.	O
With	O
development	O
and	O
test	O
F	O
1	O
scores	O
of	O
83.7	O
%	O
and	O
75.8	O
%	O
,	O
respectively	O
,	O
our	O
event	O
tagger	O
slightly	O
outperforms	O
the	O
best	O
performing	O
model	O
in	O
Sims	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
which	O
reached	O
73.9	O
%	O
F	O
1	O
.	O
In	O
our	O
analyses	O
,	O
we	O
use	O
our	O
tagger	O
to	O
detect	O
the	O
number	O
of	O
realis	O
event	O
mentions	O
.	O

We	O
design	O
a	O
commonsense	O
extraction	O
tool	O
that	O
aligns	O
sentences	O
in	O
stories	O
with	O
commonsense	O
tuples	O
,	O
using	O
a	O
heuristic	O
matching	O
algorithm	O
.	O
Given	O
a	O
story	O
,	O
we	O
match	O
possible	O
ATOMIC	O
events	O
to	O
sentences	O
by	O
selecting	O
events	O
that	O
share	O
noun	O
chunks	O
and	O
verb	O
phrases	O
with	O
sentences	O
.	O
For	O
every	O
sentence	O
s	O
i	O
that	O
matches	O
an	O
event	O
E	O
in	O
ATOMIC	O
,	O
we	O
check	O
surrounding	O
sentences	O
for	O
mentions	O
of	O
commonsense	O
inferences	O
(	O
using	O
the	O
same	O
noun	O
and	O
verb	O
phrase	O
matching	O
strategy	O
)	O
;	O
specifically	O
,	O
we	O
check	O
the	O
n	O
c	O
preceding	O
sentences	O
for	O
matches	O
of	O
causes	O
of	O
E	O
,	O
and	O
the	O
n	O
e	O
following	O
sentences	O
for	O
event	O
E	O
's	O
effects	O
.	O

To	O
measure	O
the	O
prevalence	O
of	O
semantic	O
memory	O
in	O
a	O
story	O
,	O
we	O
count	O
the	O
number	O
of	O
sentences	O
that	O
matched	O
ATOMIC	O
knowledge	O
tuples	O
in	O
their	O
surrounding	O
context	O
.	O
We	O
use	O
a	O
context	O
window	O
of	O
size	O
n	O
c	O
=	O
n	O
e	O
=	O
2	O
to	O
match	O
inferences	O
,	O
and	O
use	O
the	O
spaCy	O
pipeline	O
(	O
Honnibal	O
and	O
Montani	O
,	O
2017	O
)	O
to	O
extract	O
noun	O
and	O
verb	O
phrases	O
.	O

C.1	O
Linearity	O
with	O
Varying	O
Context	O
Size	O
Shown	O
in	O
Figure	O
5	O
,	O
we	O
compare	O
the	O
negative	O
loglikelihood	O
of	O
sentences	O
when	O
conditioned	O
on	O
varying	O
history	O
sizes	O
(	O
using	O
the	O
story	O
summary	O
as	O
context	O
E	O
)	O
.	O
As	O
expected	O
,	O
conditioning	O
on	O
longer	O
histories	O
increases	O
the	O
predictability	O
of	O
a	O
sentence	O
.	O
However	O
,	O
this	O
effect	O
is	O
significantly	O
larger	O
for	O
imagined	O
stories	O
,	O
which	O
suggests	O
that	O
imagined	O
stories	O
flow	O
more	O
linearly	O
than	O
recalled	O
stories	O
.	O

Multilingual	O
BERT	O
Post	O
-	O
Pretraining	O
Alignment	O

We	O
propose	O
a	O
simple	O
method	O
to	O
align	O
multilingual	O
contextual	O
embeddings	O
as	O
a	O
postpretraining	O
step	O
for	O
improved	O
cross	O
-	O
lingual	O
transferability	O
of	O
the	O
pretrained	O
language	O
models	O
.	O
Using	O
parallel	O
data	O
,	O
our	O
method	O
aligns	O
embeddings	O
on	O
the	O
word	O
level	O
through	O
the	O
recently	O
proposed	O
Translation	O
Language	O
Modeling	O
objective	O
as	O
well	O
as	O
on	O
the	O
sentence	O
level	O
via	O
contrastive	O
learning	O
and	O
random	O
input	O
shuffling	O
.	O
We	O
also	O
perform	O
sentence	O
-	O
level	O
code	O
-	O
switching	O
with	O
English	O
when	O
finetuning	O
on	O
downstream	O
tasks	O
.	O
On	O
XNLI	O
,	O
our	O
best	O
model	O
(	O
initialized	O
from	O
mBERT	O
)	O
improves	O
over	O
mBERT	O
by	O
4.7	O
%	O
in	O
the	O
zero	O
-	O
shot	O
setting	O
and	O
achieves	O
comparable	O
result	O
to	O
XLM	O
for	O
translate	O
-	O
train	O
while	O
using	O
less	O
than	O
18	O
%	O
of	O
the	O
same	O
parallel	O
data	O
and	O
31	O
%	O
fewer	O
model	O
parameters	O
.	O
On	O
MLQA	O
,	O
our	O
model	O
outperforms	O
XLM	O
-	O
R	O
Base	O
,	O
which	O
has	O
57	O
%	O
more	O
parameters	O
than	O
ours	O
.	O

Introduction	O

Building	O
on	O
the	O
success	O
of	O
monolingual	O
pretrained	O
language	O
models	O
(	O
LM	O
)	O
such	O
as	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
RoBERTa	O
(	O
Liu	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
their	O
multilingual	O
counterparts	O
mBERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
XLM	O
-	O
R	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2020	O
)	O
are	O
trained	O
using	O
the	O
same	O
objectives	O
-	O
Masked	O
Language	O
Modeling	O
(	O
MLM	O
)	O
and	O
in	O
the	O
case	O
of	O
mBERT	O
,	O
Next	O
Sentence	O
Prediction	O
(	O
NSP	O
)	O
.	O
MLM	O
is	O
applied	O
to	O
monolingual	O
text	O
that	O
covers	O
over	O
100	O
languages	O
.	O
Despite	O
the	O
absence	O
of	O
parallel	O
data	O
and	O
explicit	O
alignment	O
signals	O
,	O
these	O
models	O
transfer	O
surprisingly	O
well	O
from	O
high	O
resource	O
languages	O
,	O
such	O
as	O
English	O
,	O
to	O
other	O
languages	O
.	O
On	O
the	O
Natural	O
Language	O
Inference	O
(	O
NLI	O
)	O
task	O
XNLI	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
a	O
text	O
classification	O
model	O
trained	O
on	O
English	O
training	O
data	O
can	O
be	O
directly	O
applied	O
to	O
the	O
other	O
14	O
languages	O
and	O
achieve	O
respectable	O
performance	O
.	O
Having	O
a	O
single	O
model	O
that	O
can	O
serve	O
over	O
100	O
languages	O
also	O
has	O
important	O
business	O
applications	O
.	O

Recent	O
work	O
improves	O
upon	O
these	O
pretrained	O
models	O
by	O
adding	O
cross	O
-	O
lingual	O
tasks	O
leveraging	O
parallel	O
data	O
that	O
always	O
involve	O
English	O
.	O
Conneau	O
and	O
Lample	O
(	O
2019	O
)	O
pretrain	O
a	O
new	O
Transformerbased	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
model	O
from	O
scratch	O
with	O
an	O
MLM	O
objective	O
on	O
monolingual	O
data	O
,	O
and	O
a	O
Translation	O
Language	O
Modeling	O
(	O
TLM	O
)	O
objective	O
on	O
parallel	O
data	O
.	O
Cao	O
et	O
al	O
.	O
(	O
2020	O
)	O
align	O
mBERT	O
embeddings	O
in	O
a	O
post	O
-	O
hoc	O
manner	O
:	O
They	O
first	O
apply	O
a	O
statistical	O
toolkit	O
,	O
FastAlign	O
(	O
Dyer	O
et	O
al	O
.	O
,	O
2013	O
)	O
,	O
to	O
create	O
word	O
alignments	O
on	O
parallel	O
sentences	O
.	O
Then	O
,	O
mBERT	O
is	O
tuned	O
via	O
minimizing	O
the	O
mean	O
squared	O
error	O
between	O
the	O
embeddings	O
of	O
English	O
words	O
and	O
those	O
of	O
the	O
corresponding	O
words	O
in	O
other	O
languages	O
.	O
Such	O
post	O
-	O
hoc	O
approach	O
suffers	O
from	O
the	O
limitations	O
of	O
word	O
-	O
alignment	O
toolkits	O
:	O
(	O
1	O
)	O
the	O
noises	O
from	O
FastAlign	O
can	O
lead	O
to	O
error	O
propagation	O
to	O
the	O
rest	O
of	O
the	O
pipeline	O
;	O
(	O
2	O
)	O
FastAlign	O
mainly	O
creates	O
the	O
alignments	O
with	O
word	O
-	O
level	O
translation	O
and	O
usually	O
overlooks	O
the	O
contextual	O
semantic	O
compositions	O
.	O
As	O
a	O
result	O
,	O
the	O
tuned	O
mBERT	O
is	O
biased	O
to	O
shallow	O
cross	O
-	O
lingual	O
correspondence	O
.	O
Importantly	O
,	O
both	O
approaches	O
only	O
involve	O
word	O
-	O
level	O
alignment	O
tasks	O
.	O

Method	O

This	O
section	O
introduces	O
our	O
proposed	O
Post	O
-	O
Pretraining	O
Alignment	O
(	O
PPA	O
)	O
method	O
.	O
We	O
first	O
describe	O
the	O
MoCo	O
contrastive	O
learning	O
framework	O
and	O
how	O
we	O
use	O
it	O
for	O
sentence	O
-	O
level	O
alignment	O
.	O
Next	O
,	O
we	O
describe	O
the	O
finer	O
-	O
grained	O
word	O
-	O
level	O
alignment	O
with	O
TLM	O
.	O
Finally	O
,	O
when	O
training	O
data	O
in	O
the	O
target	O
language	O
is	O
available	O
,	O
we	O
incorporate	O
sentence	O
-	O
level	O
code	O
-	O
switching	O
as	O
a	O
form	O
of	O
both	O
alignment	O
and	O
data	O
augmentation	O
to	O
complement	O
PPA	O
.	O
Figure	O
1	O
shows	O
our	O
overall	O
model	O
structure	O
.	O

Background	O
:	O
Contrastive	O
Learning	O
Instance	O
discrimination	O
-	O
based	O
contrastive	O
learning	O
aims	O
to	O
bring	O
two	O
views	O
of	O
the	O
same	O
source	O
image	O
closer	O
to	O
each	O
other	O
in	O
the	O
representation	O
space	O
while	O
encouraging	O
views	O
of	O
different	O
source	O
images	O
to	O
be	O
dissimilar	O
through	O
a	O
contrastive	O
loss	O
.	O
Recent	O
advances	O
in	O
this	O
area	O
,	O
such	O
as	O
SimCLR	O
(	O
Chen	O
et	O
al	O
.	O
,	O
2020	O
)	O
and	O
MoCo	O
(	O
He	O
et	O
al	O
.	O
,	O
2020	O
)	O
have	O
bridged	O
the	O
gap	O
in	O
performance	O
between	O
self	O
-	O
supervised	O
representation	O
learning	O
and	O
fully	O
-	O
supervised	O
methods	O
on	O
the	O
ImageNet	O
(	O
Deng	O
et	O
al	O
.	O
,	O
2009	O
)	O
dataset	O
.	O
As	O
a	O
key	O
feature	O
for	O
both	O
methods	O
,	O
a	O
large	O
number	O
of	O
negative	O
examples	O
per	O
instance	O
are	O
necessary	O
for	O
the	O
models	O
to	O
learn	O
such	O
good	O
representations	O
.	O
SimCLR	O
uses	O
in	O
-	O
batch	O
negative	O
example	O
sampling	O
,	O
thus	O
requiring	O
a	O
large	O
batch	O
size	O
,	O
whereas	O
MoCo	O
stores	O
negative	O
examples	O
in	O
a	O
queue	O
and	O
casts	O
the	O
contrastive	O
learning	O
task	O
as	O
dictionary	O
(	O
query	O
-	O
key	O
)	O
lookup	O
.	O
In	O
what	O
follows	O
,	O
we	O
first	O
describe	O
MoCo	O
and	O
then	O
how	O
we	O
use	O
it	O
for	O
sentence	O
-	O
level	O
alignment	O
.	O

θ	O
k	O
=	O
mθ	O
k	O
+	O
(	O
1	O
−	O
m)θ	O
q	O
(	O
1	O
)	O

where	O
θ	O
q	O
and	O
θ	O
k	O
are	O
model	O
parameters	O
of	O
f	O
q	O
and	O
f	O
k	O
,	O
respectively	O
.	O
m	O
is	O
the	O
momentum	O
coefficient	O
.	O

Sentence	O
-	O
Level	O
Alignment	O
Objective	O

Our	O
sentence	O
-	O
level	O
alignment	O
falls	O
under	O
the	O
general	O
problem	O
of	O
bringing	O
two	O
views	O
of	O
inputs	O
from	O
the	O
same	O
source	O
closer	O
in	O
the	O
representation	O
space	O
while	O
keeping	O
those	O
from	O
different	O
sources	O
dissimilar	O
through	O
a	O
contrastive	O
loss	O
.	O
From	O
a	O
crosslingual	O
alignment	O
perspective	O
,	O
we	O
treat	O
an	O
English	O
sequence	O
S	O
en	O
i	O
and	O
its	O
translation	O
S	O
tr	O
i	O
in	O
another	O
language	O
tr	O
∈	O
L	O
as	O
two	O
manifestations	O
of	O
the	O
same	O
semantics	O
.	O
At	O
the	O
same	O
time	O
,	O
sentences	O
that	O
are	O
not	O
translations	O
of	O
each	O
other	O
should	O
be	O
further	O
apart	O
in	O
the	O
representation	O
space	O
.	O
Given	O
parallel	O
corpora	O
consisting	O
of	O
{	O
(	O
S	O
en	O
1	O
,	O
S	O
tr	O
1	O
)	O
,	O
.	O
.	O
.	O
,	O
(	O
S	O
en	O
N	O
,	O
S	O
tr	O
N	O
)	O
}	O
,	O
we	O
align	O
sentence	O
representations	O
in	O
all	O
the	O
different	O
languages	O
together	O
using	O
MoCo	O
.	O

We	O
use	O
the	O
pretrained	O
mBERT	O
model	O
to	O
initialize	O
both	O
the	O
query	O
and	O
momentum	O
encoders	O
.	O
mBERT	O
is	O
made	O
of	O
12	O
Transformer	O
blocks	O
,	O
12	O
attention	O
heads	O
,	O
and	O
hidden	O
size	O
d	O
h	O
=	O
768	O
.	O
For	O
input	O
,	O
instead	O
of	O
feeding	O
the	O
query	O
encoder	O
with	O
English	O
examples	O
and	O
the	O
momentum	O
encoder	O
with	O
translation	O
examples	O
or	O
vice	O
versa	O
,	O
we	O
propose	O
a	O
random	O
input	O
shuffling	O
approach	O
.	O
Specifically	O
,	O
we	O
randomly	O
shuffle	O
the	O
order	O
of	O
S	O
en	O
i	O
and	O
S	O
tr	O
i	O
when	O
feeding	O
the	O
two	O
encoders	O
,	O
so	O
that	O
the	O
query	O
encoder	O
sees	O
both	O
English	O
and	O
translation	O
examples	O
.	O
We	O
observe	O
that	O
this	O
is	O
a	O
crucial	O
step	O
towards	O
learning	O
good	O
multilingual	O
representations	O
using	O
our	O
method	O
.	O
The	O
final	O
hidden	O
state	O
h	O
∈	O
R	O
1×d	O
h	O
of	O
the	O
[	O
CLS	O
]	O
token	O
,	O
normalized	O
with	O
L	O
2	O
norm	O
,	O
is	O
treated	O
as	O
the	O
sentence	O
representation	O
1	O
.	O
Following	O
Chen	O
et	O
al	O
.	O
(	O
2020	O
)	O
,	O
we	O
add	O
a	O
non	O
-	O
linear	O
projection	O
layer	O
on	O
top	O
of	O
h	O
:	O

z	O
=	O
W	O
2	O
ReLU	O
(	O
W	O
1	O
h),(2	O
)	O

where	O

W	O
1	O
∈	O
R	O
d	O
h	O
×d	O
h	O
,	O
W	O
2	O
∈	O
R	O
d	O
k	O
×d	O
h	O

,	O
and	O
d	O
k	O
is	O
set	O
to	O
300	O
.	O
The	O
model	O
is	O
trained	O
using	O
the	O
InfoNCE	O
loss	O
:	O

L	O
MoCo	O
=	O
−	O
log	O
exp(z	O
q	O
•	O
z	O
k+	O
/τ	O
)	O
K	O
k=1	O
exp(z	O
q	O
•	O
z	O
k	O
/τ	O
)	O
,	O
(	O
3	O
)	O

where	O
τ	O
is	O
a	O
temperature	O
parameter	O
.	O
In	O
our	O
implementation	O
,	O
we	O
use	O
a	O
relatively	O
small	O
batch	O
size	O
of	O
128	O
,	O
resulting	O
in	O
more	O
frequent	O
parameter	O
updates	O
than	O
if	O
a	O
large	O
batch	O
size	O
were	O
used	O
.	O
Items	O
enqueued	O
early	O
on	O
can	O
thus	O
become	O
outdated	O
with	O
a	O
large	O
queue	O
,	O
so	O
we	O
scale	O
down	O
the	O
queue	O
size	O
to	O
K	O
=	O
32	O
,	O
000	O
to	O
prevent	O
the	O
queue	O
from	O
becoming	O
stale	O
.	O

Word	O
-	O
Level	O
Alignment	O
Objective	O

We	O
use	O
TLM	O
for	O
word	O
-	O
level	O
alignment	O
.	O
TLM	O
is	O
an	O
extension	O
of	O
MLM	O
that	O
operates	O
on	O
bilingual	O
data	O
-	O
parallel	O
sentences	O
are	O
concatenated	O
and	O
MLM	O
is	O
applied	O
to	O
the	O
combined	O
bilingual	O
sequence	O
.	O
Different	O
from	O
Conneau	O
and	O
Lample	O
(	O
2019	O
)	O
,	O
we	O
do	O
not	O
reset	O
positional	O
embeddings	O
when	O
forming	O
the	O
bilingual	O
sequence	O
,	O
and	O
we	O
also	O
do	O
not	O
use	O
language	O
embeddings	O
.	O
In	O
addition	O
,	O
the	O
order	O
of	O
S	O
en	O
i	O
and	O
S	O
tr	O
i	O
during	O
concatenation	O
is	O
determined	O
by	O
the	O
random	O
input	O
shuffling	O
from	O
the	O
sentence	O
-	O
level	O
alignment	O
step	O
and	O
we	O
add	O
a	O
[	O
SEP	O
]	O
token	O
between	O
S	O
en	O
i	O
and	O
S	O
tr	O
i	O
.	O
We	O
randomly	O
mask	O
15	O
%	O
of	O
the	O
WordPiece	O
tokens	O
in	O
each	O
combined	O
sequence	O
.	O
Masking	O
is	O
done	O
by	O
using	O
a	O
special	O
[	O
MASK	O
]	O
token	O
80	O
%	O
of	O
the	O
times	O
,	O
a	O
random	O
token	O
in	O
the	O
vocabulary	O
10	O
%	O
of	O
the	O
times	O
,	O
and	O
unchanged	O
for	O
the	O
remaining	O
10	O
%	O
.	O
TLM	O
is	O
performed	O
using	O
the	O
query	O
encoder	O
of	O
MoCo	O
.	O
Our	O
final	O
PPA	O
model	O
is	O
trained	O
in	O
a	O
multi	O
-	O
task	O
manner	O
with	O
both	O
sentence	O
-	O
level	O
objective	O
and	O
TLM	O
:	O

L	O
=	O
L	O
MoCo	O
+	O
L	O
TLM	O
,	O
(	O
4	O
)	O

Finetuning	O
on	O
Downstream	O
Tasks	O

After	O
an	O
alignment	O
model	O
is	O
trained	O
with	O
PPA	O
,	O
we	O
extract	O
the	O
query	O
encoder	O
from	O
MoCo	O
and	O
finetune	O
it	O
on	O
downstream	O
tasks	O
for	O
evaluation	O
.	O
We	O
follow	O
the	O
standard	O
way	O
of	O
finetuning	O
BERT	O
-	O
like	O
models	O
for	O
sequence	O
classification	O
and	O
QA	O
tasks	O
:	O

(	O
1	O
)	O
on	O
XNLI	O
,	O
we	O
concatenate	O
the	O
premise	O
with	O
the	O
hypothesis	O
,	O
and	O
add	O
a	O
[	O
SEP	O
]	O
token	O
in	O
between	O
.	O

A	O
softmax	O
classifier	O
is	O
added	O
on	O
top	O
of	O
the	O
final	O
hidden	O
state	O
of	O
the	O
[	O
CLS	O
]	O
token	O
;	O
(	O
2	O
)	O
on	O
MLQA	O
,	O
we	O
concatenate	O
the	O
question	O
with	O
the	O
context	O
,	O
and	O
add	O
a	O
[	O
SEP	O
]	O
token	O
in	O
between	O
.	O
We	O
add	O
two	O
linear	O
layers	O
on	O
top	O
of	O
mBERT	O
followed	O
by	O
softmax	O
over	O
the	O
context	O
tokens	O
to	O
predict	O
answer	O
start	O
and	O
end	O
positions	O
,	O
respectively	O
.	O
We	O
conduct	O
experiments	O
in	O
two	O
settings	O
:	O
1	O
.	O
Zeroshot	O
cross	O
-	O
lingual	O
transfer	O
,	O
where	O
training	O
data	O
is	O
available	O
in	O
English	O
but	O
not	O
in	O
target	O
languages	O
.	O
2	O
.	O
Translate	O
-	O
train	O
,	O
where	O
the	O
English	O
training	O
set	O
is	O
(	O
machine	O
)	O
translated	O
to	O
all	O
the	O
target	O
languages	O
.	O
For	O
the	O
latter	O
setting	O
,	O
we	O
perform	O
data	O
augmentation	O
with	O
code	O
-	O
switched	O
inputs	O
,	O
when	O
training	O
on	O
languages	O
other	O
than	O
English	O
.	O
For	O
example	O
,	O
a	O
Spanish	O
question	O
q	O
es	O
and	O
context	O
c	O
es	O
pair	O
can	O
be	O
augmented	O
to	O
two	O
question	O
-	O
context	O
pairs	O
(	O
q	O
es	O
,	O
c	O
en	O
)	O
and	O
(	O
q	O
en	O
,	O
c	O
es	O
)	O
with	O
code	O
-	O
switching	O
,	O
resulting	O
in	O
2x	O
training	O
data	O
2	O
.	O
The	O
same	O
goes	O
for	O
XNLI	O
with	O
premises	O
and	O
hypotheses	O
.	O
The	O
code	O
-	O
switching	O
is	O
always	O
between	O
English	O
,	O
and	O
a	O
target	O
language	O
.	O
During	O
training	O
,	O
we	O
ensure	O
the	O
two	O
augmented	O
pairs	O
appear	O
in	O
the	O
same	O
batch	O
.	O

3	O
Experimental	O
Settings	O

Parallel	O
Data	O
for	O
Post	O
-	O
Pretraining	O

Parallel	O
Data	O
All	O
parallel	O
data	O
we	O
use	O
involve	O
English	O
as	O
the	O
source	O
language	O
.	O
Specifically	O
,	O
we	O
collect	O
en	O
-	O
fr	O
,	O
en	O
-	O
es	O
,	O
en	O
-	O
de	O
parallel	O
pairs	O
from	O
Europarl	O
,	O
en	O
-	O
ar	O
,	O
en	O
-	O
zh	O
from	O
MultiUN	O
(	O
Ziemski	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
en	O
-	O
hi	O
from	O
IITB	O
(	O
Kunchukuttan	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
and	O
en	O
-	O
bg	O
from	O
both	O
Europarl	O
and	O
EUbookshop	O
.	O
All	O
datasets	O
were	O
downloaded	O
from	O
the	O
OPUS	O
3	O
website	O
(	O
Tiedemann	O
,	O
2012	O
)	O
.	O
In	O
our	O
experiments	O
,	O
we	O
vary	O
the	O
number	O
of	O
parallel	O
sentence	O
pairs	O
for	O
PPA	O
.	O
For	O
each	O
language	O
,	O
we	O
take	O
the	O
first	O
250k	O
,	O
600k	O
,	O
and	O
2	O
M	O
English	O
-	O
translation	O
parallel	O
sentence	O
pairs	O
except	O
for	O
those	O
too	O
short	O
(	O
where	O
either	O
sentence	O
has	O
less	O
than	O
10	O
WordPiece	O
tokens	O
)	O
,	O
or	O
too	O
long	O
(	O
where	O
both	O
sentences	O
concatenated	O
together	O
have	O
more	O
than	O
128	O
WordPiece	O
tokens	O
)	O
.	O
Table	O
1	O
shows	O
the	O
actual	O
number	O
of	O
parallel	O
pairs	O
in	O
each	O
of	O
our	O
250k	O
,	O
600k	O
,	O
and	O
2	O
M	O
settings	O
.	O

Evaluation	O
Benchmarks	O

XNLI	O
is	O
an	O
evaluation	O
dataset	O
for	O
cross	O
-	O
lingual	O
NLI	O
that	O
covers	O
15	O
languages	O
.	O
The	O
dataset	O
is	O
human	O
-	O
translated	O
from	O
the	O
development	O
and	O
test	O
sets	O
of	O
the	O
English	O
MultiNLI	O
dataset	O
.	O
Given	O
a	O
sentence	O
pair	O
of	O
premise	O
and	O
hypothesis	O
,	O
the	O
task	O
is	O
to	O
classify	O
their	O
relationship	O
as	O
entailment	O
,	O
contradiction	O
,	O
and	O
neutral	O
.	O
For	O
zero	O
-	O
shot	O
cross	O
-	O
lingual	O
transfer	O
,	O
we	O
train	O
on	O
the	O
English	O
MultiNLI	O
training	O
set	O
,	O
and	O
apply	O
the	O
model	O
to	O
the	O
test	O
sets	O
of	O
the	O
other	O
languages	O
.	O
For	O
translatetrain	O
,	O
we	O
train	O
on	O
translation	O
data	O
that	O
come	O
with	O
the	O
dataset	O
4	O
.	O

MLQA	O
is	O
an	O
evaluation	O
dataset	O
for	O
QA	O
that	O
covers	O
seven	O
languages	O
.	O
The	O
dataset	O
is	O
derived	O
from	O
a	O
three	O
step	O
process	O
.	O
We	O
focus	O
on	O
XLT	O
in	O
this	O
work	O
.	O
For	O
zero	O
-	O
shot	O
crosslingual	O
transfer	O
,	O
we	O
train	O
on	O
the	O
English	O
SQuAD	O
v1.1	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2016	O
)	O
training	O
set	O
.	O
For	O
translate	O
-	O
train	O
,	O
we	O
train	O
on	O
translation	O
data	O
provided	O
in	O
Hu	O
et	O
al	O
.	O
(	O
2020	O
)	O
5	O

Training	O
Details	O

Results	O

We	O
report	O
results	O
on	O
the	O
test	O
set	O
of	O
XNLI	O
and	O
MLQA	O
and	O
we	O
do	O
hyperparameter	O
searching	O
on	O
the	O
development	O
set	O
.	O
All	O
the	O
experiments	O
for	O
translatetrain	O
were	O
done	O
using	O
the	O
code	O
-	O
switching	O
technique	O
introduced	O
in	O
Section	O
2	O
.	O

XNLI	O
Table	O
2	O
shows	O
results	O
on	O
XNLI	O
measured	O
by	O
accuracy	O
.	O
Devlin	O
et	O
al	O
.	O
(	O
2019	O
)	O
only	O
provide	O
results	O
on	O
a	O
few	O
languages	O
6	O
,	O
so	O
we	O
use	O
the	O
mBERT	O
results	O
from	O
as	O
our	O
baseline	O
for	O
zeroshot	O
cross	O
-	O
lingual	O
transfer	O
,	O
and	O
Wu	O
and	O
Dredze	O
(	O
2019	O
)	O
for	O
translate	O
-	O
train	O
.	O
Our	O
best	O
model	O
,	O
trained	O
with	O
2	O
M	O
parallel	O
sentences	O
per	O
language	O
improves	O
over	O
mBERT	O
baseline	O
by	O
4.7	O
%	O
for	O
zero	O
-	O
shot	O
transfer	O
,	O
and	O
3.2	O
%	O
for	O
translate	O
-	O
train	O
.	O

Compared	O
to	O
Cao	O
et	O
al	O
.	O
(	O
2020	O
)	O
,	O
which	O
use	O
250k	O
parallel	O
sentences	O
per	O
language	O
from	O
the	O
same	O
sources	O
as	O
we	O
do	O
for	O
post	O
-	O
pretraining	O
alignment	O
,	O
our	O
250k	O
model	O
does	O
better	O
for	O
all	O
languages	O
considered	O
and	O
we	O
do	O
not	O
rely	O
on	O
the	O
word	O
-	O
to	O
-	O
word	O
pre	O
-	O
alignment	O
step	O
using	O
FastAlign	O
,	O
which	O
is	O
prone	O
to	O
error	O
propagation	O
to	O
the	O
rest	O
of	O
the	O
pipeline	O
.	O

Compared	O
to	O
XLM	O
,	O
our	O
250k	O
,	O
600k	O
and	O
2	O
M	O
settings	O
represent	O
3.1	O
%	O
,	O
7	O
%	O
and	O
17.8	O
%	O
of	O
the	O
parallel	O
data	O
used	O
by	O
XLM	O
,	O
respectively	O
(	O
see	O
Table	O
1	O
)	O
.	O
The	O
XLM	O
model	O
also	O
has	O
45	O
%	O
more	O
parameters	O
than	O
ours	O
as	O
Table	O
3	O
shows	O
.	O
Furthermore	O
,	O
XLM	O
trained	O
with	O
MLM	O
only	O
is	O
already	O
significantly	O
better	O
than	O
mBERT	O
even	O
though	O
the	O
source	O
of	O
its	O
training	O
data	O
is	O
the	O
same	O
as	O
mBERT	O
from	O
Wikipedia	O
.	O
One	O
reason	O
could	O
be	O
that	O
XLM	O
contains	O
45	O
%	O
more	O
model	O
parameters	O
than	O
mBERT	O
as	O
model	O
depth	O
and	O
capacity	O
are	O
shown	O
to	O
be	O
key	O
to	O
cross	O
-	O
lingual	O
success	O
(	O
K	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
Additionally	O
,	O
Wu	O
and	O
Dredze	O
(	O
2019	O
)	O
hypothesize	O
that	O
limiting	O
pretraining	O
to	O
the	O
languages	O
used	O
by	O
downstream	O
tasks	O
may	O
be	O
beneficial	O
since	O
XLM	O
models	O
are	O
pretrained	O
on	O
the	O
15	O
XNLI	O
languages	O
only	O
.	O
Our	O
2	O
M	O
model	O
bridges	O
the	O
gap	O
between	O
mBERT	O
and	O
XLM	O
from	O
7.5	O
%	O
to	O
2.8	O
%	O
for	O
zero	O
-	O
shot	O
transfer	O
.	O
Note	O
that	O
,	O
for	O
bg	O
,	O
our	O
total	O
processed	O
pool	O
of	O
en	O
-	O
bg	O
data	O
consists	O
of	O
456k	O
parallel	O
sentences	O
,	O
so	O
there	O
is	O
no	O
difference	O
in	O
en	O
-	O
bg	O
data	O
between	O
our	O
600k	O
and	O
2	O
M	O
settings	O
.	O
For	O
translatetrain	O
,	O
our	O
model	O
achieves	O
comparable	O
performance	O
to	O
XLM	O
with	O
the	O
further	O
help	O
of	O
code	O
-	O
switching	O
during	O
finetuning	O
.	O

Our	O
alignment	O
-	O
oriented	O
method	O
is	O
,	O
to	O
a	O
large	O
degree	O
,	O
upper	O
-	O
bounded	O
by	O
the	O
English	O
performance	O
,	O
since	O
all	O
our	O
parallel	O
data	O
involve	O
English	O
and	O
all	O
the	O
other	O
languages	O
are	O
implicitly	O
aligning	O
with	O
English	O
through	O
our	O
PPA	O
objectives	O
.	O
Our	O
2	O
M	O
model	O
is	O
able	O
to	O
improve	O
the	O
English	O
performance	O
to	O
82.4	O
from	O
the	O
mBERT	O
baseline	O
,	O
but	O
it	O
is	O
still	O
lower	O
than	O
XLM	O
(	O
MLM	O
)	O
,	O
and	O
much	O
lower	O
than	O
XLM	O
(	O
MLM+TLM	O
)	O
.	O
We	O
hypothesize	O
that	O
more	O
highquality	O
monolingual	O
data	O
and	O
model	O
capacity	O
are	O
needed	O
to	O
further	O
improve	O
our	O
English	O
performance	O
,	O
thereby	O
helping	O
other	O
languages	O
better	O
align	O
with	O
it	O
.	O

MLQA	O
Table	O
4	O
shows	O
results	O
on	O
MLQA	O
measured	O
by	O
F1	O
score	O
.	O
We	O
notice	O
the	O
mBERT	O
baseline	O
from	O
the	O
original	O
MLQA	O
paper	O
is	O
significantly	O
lower	O
than	O
that	O
from	O
,	O
so	O
we	O
use	O
the	O
latter	O
as	O
our	O
baseline	O
.	O
Our	O
2	O
M	O
model	O
outperforms	O
the	O
baseline	O
by	O
2.3	O
%	O
for	O
zero	O
-	O
shot	O
and	O
is	O
also	O
0.2	O
%	O
better	O
than	O
XLM	O
-	O
R	O
Base	O
,	O
which	O
uses	O
57	O
%	O
more	O
model	O
parameters	O
than	O
mBERT	O
as	O
Table	O
3	O
shows	O
.	O
For	O
translate	O
-	O
train	O
,	O
our	O
250k	O
model	O
is	O
1.3	O
%	O
better	O
than	O
the	O
baseline	O
.	O

Comparing	O
our	O
model	O
performance	O
using	O
vary-	O
et	O
al	O
.	O
(	O
2020	O
)	O
.	O
L	O
is	O
the	O
number	O
of	O
Transformer	O
layers	O
,	O
H	O
m	O
is	O
the	O
hidden	O
size	O
,	O
H	O
f	O
f	O
is	O
the	O
dimension	O
of	O
the	O
feed	O
-	O
forward	O
layer	O
,	O
A	O
is	O
the	O
number	O
of	O
attention	O
heads	O
,	O
and	O
V	O
is	O
the	O
vocabulary	O
size	O
.	O

ing	O
amounts	O
of	O
parallel	O
data	O
,	O
we	O
observe	O
that	O
600k	O
per	O
language	O
is	O
our	O
sweet	O
spot	O
considering	O
the	O
trade	O
-	O
off	O
between	O
resource	O
and	O
performance	O
.	O
Going	O
up	O
to	O
2	O
M	O
helps	O
on	O
XNLI	O
,	O
but	O
less	O
significantly	O
compared	O
to	O
the	O
gain	O
going	O
from	O
250k	O
to	O
600k	O
.	O
On	O
MLQA	O
,	O
surprisingly	O
,	O
250k	O
slightly	O
outperforms	O
the	O
other	O
two	O
for	O
translate	O
-	O
train	O
.	O

Ablation	O
Table	O
5	O
shows	O
the	O
contribution	O
of	O
each	O
component	O
of	O
our	O
method	O
on	O
XNLI	O
.	O
Removing	O
TLM	O
(	O
-TLM	O
)	O
consistently	O
leads	O
to	O
about	O
1	O
%	O
accuracy	O
drop	O
across	O
the	O
board	O
,	O
showing	O
positive	O
effects	O
of	O
the	O
word	O
-	O
alignment	O
objective	O
.	O
To	O
better	O
understand	O
TLM	O
's	O
consistent	O
improvement	O
,	O
we	O
replace	O
TLM	O
with	O
MLM	O
(	O
repl	O
TLM	O
w/	O
MLM	O
)	O
,	O
where	O
we	O
treat	O
S	O
en	O
i	O
and	O
S	O
tr	O
i	O
from	O
the	O
parallel	O
corpora	O
as	O
separate	O
monolingual	O
sequences	O
and	O
perform	O
MLM	O
on	O
each	O
of	O
them	O
.	O
The	O
masking	O
scheme	O
is	O
the	O
same	O
as	O
TLM	O
described	O
in	O
Section	O
2	O
.	O
We	O
observe	O
that	O
MLM	O
does	O
not	O
bring	O
significant	O
improvement	O
.	O
This	O
confirms	O
that	O
the	O
improvement	O
of	O
TLM	O
is	O
not	O
from	O
the	O
encoders	O
being	O
trained	O
with	O
more	O
data	O
and	O
iterations	O
.	O
Instead	O
,	O
the	O
word	O
-	O
alignment	O
nature	O
of	O
TLM	O
does	O
help	O
the	O
multilingual	O
training	O
.	O

Comparing	O
our	O
model	O
without	O
word	O
-	O
level	O
alignment	O
,	O
i.e.	O
,	O
-TLM	O
,	O
to	O
the	O
baseline	O
mBERT	O
in	O
Table	O
2	O
,	O
we	O
get	O
2	O
-	O
4	O
%	O
improvement	O
in	O
the	O
zero	O
-	O
shot	O
setting	O
and	O
1	O
-	O
2	O
%	O
improvement	O
in	O
translate	O
-	O
train	O
as	O
the	O
amount	O
of	O
parallel	O
data	O
is	O
increased	O
.	O
These	O
are	O
relatively	O
large	O
improvements	O
considering	O
the	O
fact	O
that	O
only	O
sentence	O
-	O
level	O
alignment	O
is	O
used	O
.	O
This	O
also	O
conforms	O
to	O
our	O
intuition	O
that	O
sentence	O
-	O
level	O
alignment	O
is	O
a	O
good	O
fit	O
here	O
since	O
XNLI	O
is	O
a	O
sentencelevel	O
task	O
.	O

In	O
the	O
zero	O
-	O
shot	O
setting	O
,	O
removing	O
MoCo	O
(	O
-MoCo	O
)	O
performs	O
similarly	O
to	O
-TLM	O
,	O
where	O
we	O
observe	O
an	O
accuracy	O
drop	O
of	O
about	O
1	O
%	O
compared	O
to	O
our	O
full	O
system	O
.	O
In	O
translate	O
-	O
train	O
,	O
-MoCo	O
outperforms	O
-TLM	O
and	O
even	O
matches	O
the	O
full	O
system	O
performance	O
for	O
250k	O
.	O

Finally	O
,	O
we	O
show	O
ablation	O
result	O
for	O
our	O
codeswitching	O
in	O
translate	O
-	O
train	O
.	O
On	O
average	O
,	O
codeswitching	O
provides	O
an	O
additional	O
gain	O
of	O
1	O
%	O
.	O
(	O
Lewis	O
et	O
al	O
.	O
,	O
2020	O
)	O
74.9	O
54.8	O
62.2	O
68.0	O
48.8	O
61.1	O
61.6	O
XLM	O
-	O
R	O
Base	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2020	O
)	O
77.1	O
54.9	O
60.9	O
67.4	O
59.4	O
61.8	O
63.6	O
Translate	O
-	O
train	O
mBERT	O
from	O
(	O
Lewis	O
et	O
al	O
.	O
,	O
2020	O
)	O
77.7	O
51.8	O
62	O
Training	O
mBERT	O
with	O
Word	O
Alignments	O
Cao	O
et	O
al	O
.	O
(	O
2020	O
)	O
post	O
-	O
align	O
mBERT	O
embeddings	O
by	O
first	O
generating	O
word	O
alignments	O
on	O
parallel	O
sentences	O
that	O
involve	O
English	O
.	O
For	O
each	O
aligned	O
word	O
pair	O
,	O
the	O
L	O
2	O
distance	O
between	O
their	O
embeddings	O
is	O
minimized	O
to	O
train	O
the	O
model	O
.	O
In	O
order	O
to	O
maintain	O
original	O
transferability	O
to	O
downstream	O
tasks	O
,	O
a	O
regularization	O
term	O
is	O
added	O
to	O
prevent	O
the	O
target	O
language	O
embeddings	O
from	O
deviating	O
too	O
much	O
from	O
their	O
mBERT	O
initialization	O
.	O
Our	O
approach	O
post	O
-	O
aligns	O
mBERT	O
with	O
two	O
self	O
-	O
supervised	O
signals	O
from	O
parallel	O
data	O
without	O
using	O
pre	O
-	O
alignment	O
tools	O
.	O
Wang	O
et	O
al	O
.	O
(	O
2019	O
)	O
also	O
align	O
mBERT	O
em-	O
Table	O
5	O
:	O
Ablation	O
Study	O
on	O
XNLI	O
.	O
250k	O
,	O
600k	O
,	O
2	O
M	O
refer	O
to	O
the	O
maximum	O
number	O
of	O
parallel	O
sentence	O
pairs	O
per	O
language	O
used	O
in	O
PPA	O
.	O
MoCo	O
refers	O
to	O
our	O
sentence	O
-	O
level	O
alignment	O
task	O
using	O
contrastive	O
learning	O
.	O
TLM	O
refers	O
to	O
our	O
word	O
-	O
level	O
alignment	O
task	O
with	O
translation	O
language	O
modeling	O
.	O
CS	O
stands	O
for	O
code	O
-	O
switching	O
.	O
We	O
conduct	O
an	O
additional	O
study	O
repl	O
TLM	O
w/	O
MLM	O
,	O
which	O
means	O
instead	O
of	O
TLM	O
training	O
,	O
we	O
augment	O
our	O
sentence	O
-	O
level	O
alignment	O
with	O
regular	O
MLM	O
on	O
monolingual	O
text	O
.	O
This	O
ablation	O
confirms	O
that	O
the	O
TLM	O
objective	O
helps	O
because	O
of	O
its	O
word	O
alignment	O
capability	O
,	O
not	O
because	O
we	O
train	O
the	O
encoders	O
with	O
more	O
data	O
and	O
iterations	O
.	O

beddings	O
using	O
parallel	O
data	O
.	O
They	O
learn	O
a	O
linear	O
transformation	O
that	O
maps	O
a	O
word	O
embedding	O
in	O
a	O
target	O
language	O
to	O
the	O
embedding	O
of	O
the	O
aligned	O
word	O
in	O
the	O
source	O
language	O
.	O
They	O
show	O
that	O
their	O
transformed	O
embeddings	O
are	O
more	O
effective	O
on	O
zero	O
-	O
shot	O
cross	O
-	O
lingual	O
dependency	O
parsing	O
.	O

Besides	O
the	O
aforementioned	O
three	O
major	O
directions	O
,	O
Artetxe	O
and	O
Schwenk	O
(	O
2019	O
)	O
train	O
a	O
multilingual	O
sentence	O
encoder	O
on	O
93	O
languages	O
.	O
Their	O
stacked	O
BiLSTM	O
encoder	O
is	O
trained	O
by	O
first	O
generating	O
embedding	O
of	O
a	O
source	O
sentence	O
and	O
then	O
decoding	O
the	O
embedding	O
into	O
the	O
target	O
sentence	O
in	O
other	O
languages	O
.	O

Concurrent	O
to	O
our	O
work	O
,	O
Chi	O
et	O
al	O
.	O
(	O
2020	O
)	O
,	O
Feng	O
et	O
al	O
.	O
(	O
2020	O
and	O
also	O
leverage	O
variants	O
of	O
contrastive	O
learning	O
for	O
cross	O
-	O
lingual	O
alignment	O
.	O
We	O
focus	O
on	O
a	O
smaller	O
model	O
and	O
improve	O
on	O
it	O
using	O
as	O
little	O
parallel	O
data	O
as	O
possible	O
.	O
We	O
also	O
explore	O
code	O
-	O
switching	O
during	O
finetuning	O
on	O
downtream	O
tasks	O
to	O
complement	O
the	O
post	O
-	O
pretraining	O
alignment	O
objectives	O
.	O

Conclusion	O

Post	O
-	O
pretraining	O
embedding	O
alignment	O
is	O
an	O
efficient	O
means	O
of	O
improving	O
cross	O
-	O
lingual	O
transferability	O
of	O
pretrained	O
multilingual	O
LMs	O
,	O
especially	O
when	O
pretraining	O
from	O
scratch	O
is	O
not	O
feasible	O
.	O
We	O
showed	O
that	O
our	O
self	O
-	O
supervised	O
sentence	O
-	O
level	O
and	O
word	O
-	O
level	O
alignment	O
tasks	O
can	O
greatly	O
improve	O
mBERT	O
's	O
performance	O
on	O
downstream	O
tasks	O
of	O
NLI	O
and	O
QA	O
,	O
and	O
the	O
method	O
can	O
potentially	O
be	O
applied	O
to	O
improve	O
other	O
pretrained	O
multilingual	O
LMs	O
.	O

In	O
addition	O
to	O
zero	O
-	O
shot	O
cross	O
-	O
lingual	O
transfer	O
,	O
we	O
also	O
showed	O
that	O
code	O
-	O
switching	O
with	O
English	O
during	O
finetuning	O
provides	O
additional	O
alignment	O
signals	O
,	O
when	O
training	O
data	O
is	O
available	O
for	O
the	O
target	O
language	O
.	O

Aspect	O
-	O
Controlled	O
Neural	O
Argument	O
Generation	O

We	O
rely	O
on	O
arguments	O
in	O
our	O
daily	O
lives	O
to	O
deliver	O
our	O
opinions	O
and	O
base	O
them	O
on	O
evidence	O
,	O
making	O
them	O
more	O
convincing	O
in	O
turn	O
.	O
However	O
,	O
finding	O
and	O
formulating	O
arguments	O
can	O
be	O
challenging	O
.	O
In	O
this	O
work	O
,	O
we	O
present	O
the	O
Arg	O
-	O
CTRL	O
-	O
a	O
language	O
model	O
for	O
argument	O
generation	O
that	O
can	O
be	O
controlled	O
to	O
generate	O
sentence	O
-	O
level	O
arguments	O
for	O
a	O
given	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
.	O
We	O
define	O
argument	O
aspect	O
detection	O
as	O
a	O
necessary	O
method	O
to	O
allow	O
this	O
fine	O
-	O
granular	O
control	O
and	O
crowdsource	O
a	O
dataset	O
with	O
5,032	O
arguments	O
annotated	O
with	O
aspects	O
.	O
Our	O
evaluation	O
shows	O
that	O
the	O
Arg	O
-	O
CTRL	O
is	O
able	O
to	O
generate	O
high	O
-	O
quality	O
,	O
aspectspecific	O
arguments	O
,	O
applicable	O
to	O
automatic	O
counter	O
-	O
argument	O
generation	O
.	O
We	O
publish	O
the	O
model	O
weights	O
and	O
all	O
datasets	O
and	O
code	O
to	O
train	O
the	O
Arg	O
-	O
CTRL	O
.	O
1	O
Nuclear	O
reactors	O
produce	O
radioactive	O
waste	O
...	O

Introduction	O

Language	O
models	O
(	O
Bengio	O
et	O
al	O
.	O
,	O
2003	O
)	O
allow	O
to	O
generate	O
text	O
through	O
learned	O
distributions	O
of	O
a	O
language	O
and	O
have	O
been	O
applied	O
to	O
a	O
variety	O
of	O
areas	O
like	O
machine	O
translation	O
(	O
Bahdanau	O
et	O
al	O
.	O
,	O
2015	O
)	O
,	O
summarization	O
(	O
Paulus	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
or	O
dialogue	O
systems	O
(	O
Wen	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
A	O
rather	O
new	O
field	O
for	O
these	O
models	O
is	O
the	O
task	O
of	O
producing	O
text	O
with	O
argumentative	O
content	O
(	O
Wang	O
and	O
Ling	O
,	O
2016	O
)	O
.	O
We	O
believe	O
this	O
technology	O
can	O
support	O
humans	O
in	O
the	O
challenging	O
task	O
of	O
finding	O
and	O
formulating	O
arguments	O
.	O
A	O
politician	O
might	O
use	O
this	O
to	O
prepare	O
for	O
a	O
debate	O
with	O
a	O
political	O
opponent	O
or	O
for	O
a	O
press	O
conference	O
.	O
It	O
may	O
be	O
used	O
to	O
support	O
students	O
in	O
writing	O
argumentative	O
essays	O
or	O
to	O
enrich	O
one	O
-	O
sided	O
discussions	O
with	O
counter	O
-	O
arguments	O
.	O
In	O
contrast	O
to	O
retrieval	O
methods	O
,	O
generation	O
allows	O
to	O
combine	O
and	O
stylistically	O
adapt	O
text	O
(	O
e.g.	O
arguments	O
)	O
based	O
on	O
a	O
given	O
input	O
(	O
usually	O
the	O
beginning	O
of	O
a	O
sentence	O
)	O
.	O
Current	O
argument	O
generation	O
models	O
,	O
however	O
,	O
produce	O
lengthy	O
texts	O
and	O
allow	O
the	O
user	O
little	O
control	O
over	O
the	O
aspect	O
the	O
argument	O
should	O
address	O
Hua	O
and	O
Wang	O
,	O
2018	O
)	O
.	O
We	O
show	O
that	O
argument	O
generation	O
can	O
be	O
enhanced	O
by	O
allowing	O
for	O
a	O
fine	O
-	O
grained	O
control	O
and	O
limiting	O
the	O
argument	O
to	O
a	O
single	O
but	O
concise	O
sentence	O
.	O

Controllable	O
language	O
models	O
like	O
the	O
CTRL	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
allow	O
to	O
condition	O
the	O
model	O
at	O
training	O
time	O
to	O
certain	O
control	O
codes	O
.	O
At	O
inference	O
,	O
these	O
can	O
be	O
used	O
to	O
direct	O
the	O
model	O
's	O
output	O
with	O
regard	O
to	O
content	O
or	O
style	O
.	O
We	O
build	O
upon	O
this	O
architecture	O
to	O
control	O
argument	O
generation	O
based	O
solely	O
on	O
a	O
given	O
topic	O
,	O
stance	O
,	O
and	O
argument	O
aspect	O
.	O
For	O
instance	O
,	O
to	O
enforce	O
focus	O
on	O
the	O
aspect	O
of	O
cancer	O
for	O
the	O
topic	O
of	O
nuclear	O
energy	O
,	O
we	O
input	O
a	O
control	O
code	O
"	O
Nuclear	O
Energy	O
CON	O
cancer	O
"	O
that	O
creates	O
a	O
contra	O
argument	O
discussing	O
this	O
aspect	O
,	O
for	O
instance	O
:	O
"	O
Studies	O
show	O
that	O
people	O
living	O
next	O
to	O
nuclear	O
power	O
plants	O
have	O
a	O
higher	O
risk	O
of	O
developing	O
cancer	O
.	O
"	O
.	O

To	O
obtain	O
control	O
codes	O
from	O
training	O
data	O
,	O
we	O
pre	O
-	O
define	O
a	O
set	O
of	O
topics	O
to	O
retrieve	O
documents	O
for	O
and	O
rely	O
on	O
an	O
existing	O
stance	O
detection	O
model	O
to	O
classify	O
whether	O
a	O
sentence	O
argues	O
in	O
favor	O
(	O
pro	O
)	O
or	O
against	O
(	O
con	O
)	O
the	O
given	O
topic	O
(	O
Stab	O
et	O
al	O
.	O
,	O
2018a	O
)	O
.	O
Regarding	O
argument	O
aspect	O
detection	O
,	O
however	O
,	O
past	O
work	O
has	O
two	O
drawbacks	O
:	O
it	O
either	O
uses	O
simple	O
rule	O
-	O
based	O
extraction	O
of	O
verb	O
-	O
and	O
noun	O
-	O
phrases	O
(	O
Fujii	O
and	O
Ishikawa	O
,	O
2006	O
)	O
or	O
the	O
definition	O
of	O
aspects	O
is	O
based	O
on	O
target	O
-	O
concepts	O
located	O
within	O
the	O
same	O
sentence	O
(	O
Gemechu	O
and	O
Reed	O
,	O
2019	O
)	O
.	O
Aspects	O
as	O
we	O
require	O
and	O
define	O
them	O
are	O
not	O
bound	O
to	O
any	O
part	O
-	O
of	O
-	O
speech	O
tag	O
and	O
(	O
1	O
)	O
hold	O
the	O
core	O
reason	O
upon	O
which	O
the	O
conclusion	O
/	O
evidence	O
is	O
built	O
and	O
(	O
2	O
)	O
encode	O
the	O
stance	O
towards	O
a	O
general	O
but	O
not	O
necessarily	O
explicitly	O
mentioned	O
topic	O
the	O
argument	O
discusses	O
.	O
For	O
instance	O
:	O

Topic	O
:	O
Nuclear	O
Energy	O
Argument	O
:	O
Running	O
nuclear	O
reactors	O
is	O
costly	O
as	O
it	O
involves	O
long	O
-	O
time	O
disposal	O
of	O
radioactive	O
waste	O
.	O

The	O
evidence	O
of	O
this	O
argument	O
is	O
based	O
upon	O
the	O
two	O
underlined	O
aspects	O
.	O
While	O
these	O
aspects	O
encode	O
a	O
negative	O
stance	O
towards	O
the	O
topic	O
of	O
"	O
Nuclear	O
Energy	O
"	O
,	O
the	O
topic	O
itself	O
is	O
not	O
mentioned	O
explicitly	O
in	O
the	O
argument	O
.	O

Our	O
final	O
controlled	O
argument	O
generation	O
pipeline	O
(	O
see	O
Figure	O
1	O
)	O
works	O
as	O
follows	O
:	O
(	O
1	O
)	O
We	O
gather	O
several	O
million	O
documents	O
for	O
eight	O
different	O
topics	O
from	O
two	O
large	O
data	O
sources	O
.	O
All	O
sentences	O
are	O
classified	O
into	O
pro-	O
,	O
con-	O
,	O
and	O
non	O
-	O
arguments	O
.	O
We	O
detect	O
aspects	O
of	O
all	O
arguments	O
with	O
a	O
model	O
trained	O
on	O
a	O
novel	O
dataset	O
and	O
concatenate	O
arguments	O
with	O
the	O
same	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
into	O
training	O
documents	O
.	O
(	O
2	O
)	O
We	O
use	O
the	O
collected	O
classified	O
data	O
to	O
condition	O
the	O
Arg	O
-	O
CTRL	O
on	O
the	O
topics	O
,	O
stances	O
,	O
and	O
aspects	O
of	O
all	O
gathered	O
arguments	O
.	O

(	O
3	O
)	O
At	O
inference	O
,	O
passing	O
the	O
control	O
code	O
[	O
Topic	O
]	O
[	O
Stance	O
]	O
[	O
Aspect	O
]	O
to	O
the	O
model	O
will	O
generate	O
an	O
argument	O
that	O
follows	O
these	O
commands	O
.	O

Our	O
evaluation	O
shows	O
that	O
the	O
Arg	O
-	O
CTRL	O
is	O
able	O
to	O
produce	O
aspect	O
-	O
specific	O
,	O
high	O
-	O
quality	O
arguments	O
,	O
applicable	O
to	O
automatic	O
counter	O
-	O
argument	O
generation	O
.	O
The	O
contributions	O
are	O
as	O
follows	O
:	O
(	O
i	O
)	O
We	O
adapt	O
and	O
fine	O
-	O
tune	O
the	O
CTRL	O
for	O
aspect	O
-	O
controlled	O
neural	O
argument	O
generation	O
.	O
(	O
ii	O
)	O
We	O
show	O
that	O
detecting	O
argument	O
aspects	O
and	O
conditioning	O
the	O
generation	O
model	O
on	O
them	O
are	O
necessary	O
steps	O
to	O
control	O
the	O
model	O
's	O
training	O
process	O
and	O
its	O
perspective	O
while	O
generating	O
.	O
(	O
iii	O
)	O
We	O
propose	O
several	O
methods	O
to	O
analyze	O
and	O
evaluate	O
the	O
quality	O
of	O
(	O
controllable	O
)	O
argument	O
generation	O
models	O
.	O
(	O
iv	O
)	O
We	O
develop	O
a	O
new	O
scheme	O
to	O
annotate	O
argument	O
aspects	O
and	O
release	O
a	O
dataset	O
with	O
5,032	O
samples	O
.	O

Related	O
Work	O

Argument	O
Aspect	O
Detection	O
Early	O
work	O
by	O
Fujii	O
and	O
Ishikawa	O
(	O
2006	O
)	O
focuses	O
mainly	O
on	O
Japanese	O
and	O
restricts	O
aspects	O
to	O
noun	O
-	O
and	O
verb	O
-	O
phrases	O
,	O
extracted	O
via	O
hand	O
-	O
crafted	O
rules	O
.	O
Boltužić	O
and	O
Šnajder	O
(	O
2017	O
)	O
extract	O
noun	O
-	O
phrases	O
and	O
aggregate	O
them	O
into	O
concepts	O
to	O
analyze	O
the	O
microstructure	O
of	O
claims	O
.	O
Misra	O
et	O
al	O
.	O
(	O
2015	O
)	O
introduce	O
facets	O
as	O
low	O
level	O
issues	O
,	O
used	O
to	O
support	O
or	O
attack	O
an	O
argumentation	O
.	O
In	O
that	O
,	O
facets	O
are	O
conceptually	O
similar	O
to	O
aspects	O
,	O
but	O
not	O
explicitly	O
phrased	O
and	O
instead	O
seen	O
as	O
abstract	O
concepts	O
that	O
define	O
clusters	O
of	O
semantically	O
similar	O
text	O
-	O
spans	O
of	O
summaries	O
.	O
Bilu	O
et	O
al	O
.	O
(	O
2019	O
)	O
define	O
commonplace	O
arguments	O
that	O
are	O
valid	O
in	O
several	O
situations	O
for	O
specified	O
actions	O
(	O
e.g.	O
"	O
ban	O
"	O
)	O
and	O
topics	O
(	O
e.g.	O
"	O
smoking	O
"	O
)	O
.	O
These	O
actions	O
are	O
similar	O
to	O
aspects	O
,	O
but	O
limited	O
in	O
number	O
and	O
manually	O
defined	O
.	O
Gemechu	O
and	O
Reed	O
(	O
2019	O
)	O
detect	O
,	O
amongst	O
others	O
,	O
concepts	O
and	O
aspects	O
in	O
arguments	O
with	O
models	O
trained	O
on	O
expert	O
annotations	O
.	O
However	O
,	O
in	O
their	O
definition	O
,	O
aspects	O
have	O
to	O
point	O
to	O
a	O
target	O
concept	O
mentioned	O
in	O
the	O
argument	O
.	O
In	O
our	O
definition	O
,	O
aspects	O
refer	O
to	O
a	O
general	O
topic	O
which	O
is	O
not	O
necessarily	O
part	O
of	O
the	O
sentence	O
and	O
our	O
annotation	O
scheme	O
is	O
applicable	O
by	O
non	O
-	O
experts	O
.	O

The	O
concept	O
of	O
framing	O
dimensions	O
(	O
Boydstun	O
et	O
al	O
.	O
,	O
2014	O
)	O
is	O
close	O
to	O
argument	O
aspects	O
.	O
In	O
the	O
field	O
of	O
argument	O
mining	O
,	O
Ajjour	O
et	O
al	O
.	O
(	O
2019	O
)	O
recently	O
applied	O
frames	O
to	O
label	O
argument	O
clusters	O
.	O
Yet	O
,	O
their	O
method	O
does	O
not	O
allow	O
to	O
detect	O
frames	O
.	O
Other	O
works	O
present	O
methods	O
to	O
automatically	O
label	O
sentences	O
of	O
news	O
articles	O
and	O
online	O
discussions	O
with	O
frames	O
(	O
Hartmann	O
et	O
al	O
.	O
,	O
2019;Naderi	O
and	O
Hirst	O
,	O
2017	O
)	O
.	O
These	O
methods	O
are	O
,	O
however	O
,	O
limited	O
to	O
a	O
small	O
set	O
of	O
predefined	O
frames	O
that	O
represent	O
high	O
-	O
level	O
concepts	O
.	O
Contrarily	O
,	O
we	O
operate	O
on	O
a	O
fine	O
-	O
grained	O
span	O
-	O
level	O
to	O
detect	O
aspects	O
that	O
are	O
explicitly	O
mentioned	O
in	O
arguments	O
.	O

Argument	O
Generation	O
Early	O
approaches	O
rely	O
on	O
rules	O
from	O
argumentation	O
theory	O
and	O
user	O
preference	O
models	O
(	O
Carenini	O
and	O
Moore	O
,	O
2006;Zukerman	O
et	O
al	O
.	O
,	O
1998	O
)	O
.	O
In	O
a	O
more	O
recent	O
work	O
,	O
Sato	O
et	O
al	O
.	O
(	O
2015	O
)	O
construct	O
rules	O
to	O
find	O
arguments	O
in	O
a	O
large	O
data	O
source	O
,	O
which	O
are	O
then	O
filtered	O
and	O
ordered	O
with	O
a	O
neural	O
network	O
based	O
ranker	O
.	O
Baff	O
et	O
al	O
.	O
(	O
2019	O
)	O
use	O
a	O
clustering	O
and	O
regression	O
approach	O
to	O
assemble	O
discourse	O
units	O
(	O
major	O
claims	O
,	O
pro	O
and	O
con	O
statements	O
)	O
to	O
argumentative	O
texts	O
.	O
However	O
,	O
most	O
of	O
these	O
approaches	O
rely	O
on	O
hand	O
-	O
crafted	O
features	O
and	O
do	O
not	O
generalize	O
well	O
.	O
Moreover	O
,	O
they	O
all	O
require	O
permanent	O
access	O
to	O
large	O
data	O
sources	O
and	O
are	O
not	O
able	O
to	O
generate	O
new	O
arguments	O
.	O

Recently	O
,	O
research	O
on	O
generating	O
arguments	O
with	O
language	O
models	O
gained	O
more	O
attention	O
.	O
use	O
a	O
sequence	O
to	O
sequence	O
model	O
(	O
Sutskever	O
et	O
al	O
.	O
,	O
2014	O
)	O
to	O
generate	O
argumentative	O
text	O
by	O
attending	O
to	O
the	O
input	O
and	O
keyphrases	O
automatically	O
extracted	O
for	O
the	O
input	O
from	O
,	O
for	O
example	O
,	O
Wikipedia	O
.	O
Other	O
work	O
focuses	O
on	O
generating	O
argumentative	O
dialogue	O
(	O
Le	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
counterarguments	O
(	O
Hidey	O
and	O
McKeown	O
,	O
2019	O
;	O
based	O
on	O
a	O
given	O
input	O
sentence	O
,	O
or	O
on	O
generating	O
summaries	O
from	O
a	O
set	O
of	O
arguments	O
(	O
Wang	O
and	O
Ling	O
,	O
2016	O
)	O
.	O
Contrarily	O
,	O
we	O
train	O
a	O
language	O
model	O
that	O
does	O
not	O
require	O
a	O
sentence	O
-	O
level	O
input	O
for	O
generation	O
and	O
allows	O
for	O
direct	O
control	O
over	O
the	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
of	O
the	O
produced	O
argument	O
.	O
Xing	O
et	O
al	O
.	O
(	O
2017	O
)	O
design	O
a	O
language	O
model	O
that	O
attends	O
to	O
topic	O
information	O
to	O
generate	O
responses	O
for	O
chatbots	O
.	O
Dathathri	O
et	O
al	O
.	O
(	O
2019	O
)	O
train	O
two	O
models	O
that	O
control	O
the	O
sentiment	O
and	O
topic	O
of	O
the	O
output	O
of	O
pre	O
-	O
trained	O
language	O
models	O
at	O
inference	O
.	O
Gretz	O
et	O
al	O
.	O
(	O
2020a	O
)	O
fine	O
-	O
tune	O
GPT-2	O
on	O
existing	O
,	O
labeled	O
datasets	O
to	O
generate	O
claims	O
for	O
given	O
topics	O
.	O
However	O
,	O
the	O
latter	O
works	O
do	O
not	O
explore	O
generation	O
for	O
such	O
a	O
fine	O
-	O
grained	O
and	O
explicit	O
control	O
as	O
proposed	O
in	O
this	O
work	O
.	O
We	O
show	O
that	O
argument	O
generation	O
requires	O
the	O
concept	O
of	O
argument	O
aspects	O
to	O
shape	O
the	O
produced	O
argument	O
's	O
perspective	O
and	O
to	O
allow	O
for	O
diverse	O
arguments	O
for	O
a	O
topic	O
of	O
interest	O
.	O

Argument	O
Aspect	O
Detection	O

Argument	O
aspect	O
detection	O
is	O
necessary	O
for	O
our	O
argument	O
generation	O
pipeline	O
,	O
as	O
it	O
allows	O
for	O
a	O
finegrained	O
control	O
over	O
the	O
generation	O
process	O
.	O
We	O
create	O
a	O
new	O
dataset	O
,	O
as	O
existing	O
approaches	O
either	O
rely	O
on	O
coarse	O
-	O
grained	O
frames	O
or	O
can	O
not	O
be	O
applied	O
by	O
non	O
-	O
expert	O
annonators	O
in	O
a	O
scalable	O
manner	O
.	O

Dataset	O
Creation	O

We	O
base	O
our	O
new	O
aspect	O
detection	O
dataset	O
on	O
the	O
UKP	O
Sentential	O
Argument	O
Mining	O
Corpus	O
(	O
UKP	O
-	O
Corpus	O
)	O
by	O
Stab	O
et	O
al	O
.	O
(	O
2018b	O
)	O
,	O
as	O
it	O
already	O
contains	O
sentence	O
-	O
level	O
arguments	O
and	O
two	O
of	O
the	O
control	O
codes	O
we	O
aim	O
to	O
use	O
:	O
topics	O
and	O
stance	O
labels	O
.	O
More	O
precisely	O
,	O
it	O
contains	O
25,474	O
manually	O
labelled	O
sentences	O
for	O
eight	O
controversial	O
topics	O
in	O
English	O
.	O
Each	O
sample	O
consists	O
of	O
a	O
topic	O
and	O
a	O
sentence	O
,	O
labelled	O
as	O
either	O
being	O
supporting	O
,	O
attacking	O
,	O
or	O
no	O
argument	O
towards	O
the	O
given	O
topic	O
.	O
As	O
we	O
are	O
only	O
interested	O
in	O
arguments	O
,	O
we	O
do	O
not	O
consider	O
the	O
non	O
-	O
argumentative	O
sentences	O
.	O

Step	O
1	O
:	O
Preliminary	O
annotations	O
To	O
ensure	O
the	O
feasibility	O
of	O
creating	O
a	O
dataset	O
for	O
this	O
task	O
,	O
two	O
experts	O
(	O
a	O
post	O
-	O
doctoral	O
researcher	O
and	O
an	O
undergraduate	O
student	O
with	O
NLP	O
background	O
)	O
independently	O
annotate	O
800	O
random	O
samples	O
(	O
from	O
four	O
topics	O
,	O
200	O
per	O
topic	O
)	O
taken	O
from	O
the	O
UKP	O
-	O
Corpus	O
.	O
The	O
annotations	O
are	O
binary	O
and	O
on	O
token	O
-	O
level	O
,	O
where	O
multiple	O
spans	O
of	O
tokens	O
could	O
be	O
selected	O
as	O
aspects	O
.	O
The	O
resulting	O
inter	O
-	O
annotator	O
agreement	O
of	O
this	O
study	O
is	O
Krippendorff	O
's	O
α	O
u	O
=	O
.38	O
.	O
While	O
this	O
shows	O
that	O
the	O
task	O
is	O
generally	O
feasible	O
,	O
the	O
agreement	O
on	O
exact	O
token	O
spans	O
is	O
rather	O
low	O
.	O
Hence	O
,	O
in	O
the	O
following	O
steps	O
,	O
we	O
reduce	O
the	O
complexity	O
of	O
the	O
annotation	O
task	O
.	O

Step	O
2	O
:	O
Annotation	O
scheme	O
Instead	O
of	O
free	O
spanlevel	O
annotations	O
,	O
we	O
present	O
annotators	O
with	O
a	O
ranked	O
list	O
of	O
aspect	O
recommendations	O
.	O
To	O
generate	O
meaningful	O
recommendations	O
,	O
we	O
train	O
a	O
ranking	O
model	O
using	O
the	O
preliminary	O
annotations	O
(	O
Step	O
1	O
)	O
.	O

Step	O
2a	O
:	O
Data	O
preparation	O
for	O
ranking	O
To	O
create	O
training	O
data	O
for	O
the	O
ranker	O
,	O
we	O
use	O
a	O
simple	O
heuristic	O
to	O
calculate	O
scores	O
between	O
0	O
and	O
1	O
for	O
all	O
N	O
-	O
grams	O
of	O
a	O
sentence	O
by	O
dividing	O
the	O
number	O
of	O
aspect	O
tokens	O
within	O
an	O
N	O
-	O
gram	O
by	O
its	O
length	O
N	O
:	O
#	O
aspect	O
tokens	O
N	O
∈	O
[	O
0	O
,	O
1	O
]	O
.	O
Our	O
analysis	O
reveals	O
that	O
96	O
%	O
(	O
783	O
of	O
814	O
)	O
of	O
all	O
aspects	O
in	O
the	O
preliminary	O
annotation	O
dataset	O
only	O
contain	O
one	O
to	O
four	O
tokens	O
.	O
We	O
thus	O
decide	O
to	O
ignore	O
all	O
candidates	O
with	O
more	O
than	O
four	O
tokens	O
.	O
No	O
other	O
limitations	O
or	O
filtering	O
mechanisms	O
are	O
applied	O
.	O

Step	O
2b	O
:	O
Training	O
the	O
ranker	O
We	O
use	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
MT	O
-	O
DNN	O
2	O
(	O
base	O
and	O
large	O
)	O
to	O
train	O
a	O
ranker	O
.	O
For	O
training	O
,	O
we	O
create	O
five	O
splits	O
:	O
(	O
1	O
)	O
one	O
in	O
-	O
topic	O
split	O
using	O
a	O
random	O
subset	O
from	O
all	O
four	O
topics	O
and	O
(	O
2	O
)	O
four	O

Topic	O

Five	O
most	O
frequent	O
aspects	O
(	O
frequency	O
)	O
Gun	O
control	O
right	O
(	O
30	O
)	O
,	O
protect	O
(	O
18	O
)	O
,	O
background	O
checks	O
(	O
17	O
)	O
,	O
gun	O
violence	O
(	O
14	O
)	O
,	O
criminal	O
(	O
13	O
)	O
Death	O
penalty	O
cost	O
(	O
16	O
)	O
,	O
innocent	O
(	O
12	O
)	O
,	O
retribution	O
(	O
10	O
)	O
,	O
murder	O
rate	O
(	O
9	O
)	O
,	O
deterrent	O
(	O
8)	O
Abortion	O
right	O
(	O
21	O
)	O
,	O
pain	O
(	O
10	O
)	O
,	O
choice	O
(	O
10	O
)	O
,	O
right	O
to	O
life	O
(	O
9	O
)	O
,	O
risk	O
(	O
9	O
)	O
Marijuana	O
legalization	O
dangerous	O
(	O
16	O
)	O
,	O
cost	O
(	O
13	O
)	O
,	O
risk	O
(	O
12	O
)	O
,	O
harm	O
(	O
10	O
)	O
,	O
black	O
market	O
(	O
9	O
)	O
General	O
aspects	O
dangerous	O
(	O
in	O
8	O
of	O
8	O
topics	O
)	O
,	O
cost	O
/	O
life	O
/	O
risk	O
/	O
safety	O
(	O
in	O
7	O
of	O
8	O
topics	O
)	O
cross	O
-	O
topic	O
splits	O
using	O
a	O
leave	O
-	O
one	O
-	O
topic	O
-	O
out	O
strategy	O
.	O
The	O
cross	O
-	O
topic	O
setup	O
allows	O
us	O
to	O
estimate	O
the	O
ranker	O
's	O
performance	O
on	O
unseen	O
topics	O
of	O
the	O
UKP	O
-	O
Corpus	O
.	O

A	O
single	O
data	O
sample	O
is	O
represented	O
by	O
an	O
argument	O
and	O
an	O
1	O
-	O
to	O
4	O
-	O
gram	O
of	O
this	O
argument	O
,	O
separated	O
by	O
the	O
BERT	O
architecture	O
's	O
[	O
SEP	O
]	O
token	O
.	O
This	O
technique	O
expands	O
the	O
800	O
original	O
samples	O
of	O
the	O
dataset	O
to	O
around	O
80,336	O
.	O
The	O
model	O
is	O
trained	O
for	O
5	O
epochs	O
,	O
with	O
a	O
learning	O
rate	O
of	O
5	O
×	O
10	O
−5	O
,	O
and	O
a	O
batch	O
size	O
of	O
8	O
.	O
We	O
use	O
the	O
mean	O
squared	O
error	O
as	O
loss	O
and	O
take	O
the	O
recall@k	O
to	O
compare	O
the	O
models	O
.	O
The	O
in	O
-	O
and	O
cross	O
-	O
topic	O
results	O
of	O
the	O
bestperforming	O
model	O
(	O
MT	O
-	O
DNN	O
BASE	O
)	O
are	O
reported	O
in	O
Table	O
2	O
.	O
All	O
results	O
are	O
the	O
average	O
over	O
runs	O
with	O
five	O
different	O
seeds	O
(	O
and	O
over	O
all	O
four	O
splits	O
for	O
the	O
cross	O
-	O
topic	O
experiments	O
)	O
.	O

Step	O
2c	O
:	O
Creating	O
the	O
annotation	O
data	O
For	O
each	O
of	O
the	O
four	O
topics	O
that	O
are	O
part	O
of	O
the	O
preliminary	O
annotation	O
dataset	O
,	O
we	O
use	O
the	O
in	O
-	O
topic	O
model	O
to	O
predict	O
aspects	O
of	O
629	O
randomly	O
chosen	O
,	O
unseen	O
arguments	O
from	O
the	O
UKP	O
-	O
Corpus	O
.	O
For	O
the	O
other	O
four	O
topics	O
of	O
the	O
UKP	O
-	O
Corpus	O
,	O
we	O
choose	O
the	O
best	O
cross	O
-	O
topic	O
model	O
to	O
predict	O
aspects	O
for	O
the	O
same	O
amount	O
of	O
samples	O
.	O
To	O
keep	O
a	O
recall	O
of	O
at	O
least	O
80	O
%	O
,	O
we	O
choose	O
the	O
ten	O
and	O
fifteen	O
highest	O
-	O
ranked	O
aspect	O
candidates	O
for	O
samples	O
as	O
predicted	O
by	O
the	O
in	O
-	O
topic	O
and	O
cross	O
-	O
topic	O
model	O
,	O
respectively	O
.	O
We	O
remove	O
aspect	O
candidates	O
that	O
include	O
punctuation	O
,	O
begin	O
or	O
end	O
with	O
stopwords	O
,	O
or	O
contain	O
digits	O
.	O

Step	O
3	O
:	O
Annotation	O
study	O
We	O
use	O
Amazon	O
Mechanical	O
Turk	O
to	O
annotate	O
each	O
sample	O
by	O
eight	O
different	O
workers	O
located	O
in	O
the	O
US	O
,	O
paying	O
$	O
7.6	O
per	O
hour	O
(	O
minimum	O
wage	O
is	O
$	O
7.25	O
per	O
hour	O
)	O
.	O
Based	O
on	O
a	O
subset	O
of	O
232	O
samples	O
,	O
we	O
compute	O
an	O
α	O
u	O
of	O
.67	O
between	O
crowdworkers	O
and	O
experts	O
(	O
three	O
doctoral	O
researchers	O
)	O
.	O
Compared	O
to	O
the	O
initial	O
study	O
,	O
the	O
new	O
approach	O
increases	O
the	O
inter	O
-	O
annotator	O
agreement	O
between	O
experts	O
by	O
approx	O
.	O
11	O
points	O
(	O
see	O
App	O
.	O
A	O
for	O
further	O
details	O
on	O
the	O
annotation	O
study	O
)	O
.	O
Based	O
on	O
this	O
promising	O
result	O
,	O
we	O
create	O
a	O
dataset	O
of	O
5,032	O
high	O
-	O
quality	O
samples	O
that	O
are	O
labelled	O
with	O
aspects	O
,	O
as	O
well	O
as	O
with	O
their	O
original	O
stance	O
labels	O
from	O
the	O
UKP	O
-	O
Corpus	O
.	O
We	O
show	O
the	O
most	O
frequent	O
(	O
lemmatized	O
)	O
aspects	O
that	O
appear	O
in	O
some	O
topics	O
in	O
Table	O
1	O
.	O

Evaluation	O

We	O
create	O
a	O
cross	O
-	O
topic	O
split	O
with	O
the	O
data	O
of	O
two	O
topics	O
as	O
test	O
set	O
(	O
gun	O
control	O
,	O
school	O
uniforms	O
)	O
,	O
one	O
topic	O
as	O
dev	O
set	O
(	O
death	O
penalty	O
)	O
,	O
and	O
the	O
remaining	O
topics	O
as	O
train	O
set	O
and	O
evaluate	O
two	O
models	O
with	O
it	O
.	O
First	O
,	O
we	O
use	O
the	O
ranking	O
approach	O
described	O
in	O
Step	O
2a-2b	O
to	O
fine	O
-	O
tune	O
MT	O
-	O
DNN	O
BASE	O
on	O
the	O
newly	O
generated	O
data	O
(	O
"	O
Ranker	O
"	O
)	O
.	O
At	O
inference	O
,	O
we	O
choose	O
the	O
top	O
T	O
aspects	O
for	O
each	O
argument	O
as	O
candidates	O
.	O
We	O
tune	O
T	O
on	O
the	O
dev	O
set	O
and	O
find	O
T	O
=	O
2	O
to	O
be	O
the	O
best	O
choice	O
.	O
Second	O
,	O
we	O
use	O
BERT	O
for	O
sequence	O
tagging	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2020	O
)	O
and	O
label	O
all	O
tokens	O
of	O
the	O
samples	O
with	O
BIO	O
tags	O
.	O
As	O
previously	O
done	O
with	O
the	O
ranker	O
,	O
we	O
experiment	O
with	O
BERT	O
and	O
MT	O
-	O
DNN	O
weights	O
and	O
find	O
BERT	O
LARGE	O
to	O
be	O
the	O
best	O
choice	O
(	O
trained	O
for	O
5	O
epochs	O
,	O
with	O
a	O
learning	O
rate	O
of	O
1	O
×	O
10	O
−5	O
and	O
a	O
batch	O
size	O
of	O
32	O
)	O
.	O
We	O
flatten	O
the	O
predictions	O
for	O
all	O
test	O
samples	O
and	O
calculate	O
the	O
F	O
1	O
,	O
Precision	O
,	O
and	O
Recall	O
macro	O
scores	O
.	O
All	O
models	O
are	O
trained	O
over	O
five	O
seeds	O
and	O
the	O
averaged	O
results	O
are	O
reported	O
in	O
Table	O
3	O
.	O

Data	O
Collection	O
Pipeline	O

This	O
section	O
describes	O
the	O
data	O
collection	O
and	O
preprocessing	O
for	O
the	O
argument	O
generation	O
pipeline	O
.	O

BERT	O
LARGE	O
predicts	O
classes	O
B	O
and	O
I	O
with	O
an	O
F	O
1	O
of	O
.65	O
and	O
.53	O
,	O
hence	O
aspects	O
with	O
more	O
than	O
one	O
token	O
are	O
less	O
well	O
identified	O
.	O
A	O
difference	O
is	O
to	O
be	O
expected	O
,	O
as	O
the	O
class	O
balance	O
of	O
B	O
's	O
to	O
I	O
's	O
is	O
2,768	O
to	O
2,103	O
.	O
While	O
the	O
ranker	O
performs	O
worse	O
based	O
on	O
the	O
shown	O
metrics	O
,	O
it	O
has	O
a	O
slightly	O
higher	O
recall	O
for	O
class	O
I.	O
We	O
assume	O
this	O
is	O
due	O
to	O
the	O
fact	O
that	O
it	O
generally	O
ranks	O
aspects	O
with	O
more	O
than	O
one	O
token	O
on	O
top	O
,	O
i.e.	O
there	O
will	O
often	O
be	O
at	O
least	O
one	O
or	O
more	O
I	O
's	O
in	O
the	O
prediction	O
.	O
In	O
contrast	O
to	O
that	O
,	O
BERT	O
LARGE	O
focuses	O
more	O
on	O
shorter	O
aspects	O
,	O
which	O
is	O
also	O
in	O
accordance	O
with	O
the	O
average	O
aspect	O
length	O
of	O
1.8	O
tokens	O
per	O
aspect	O
in	O
the	O
dataset	O
.	O
In	O
total	O
,	O
BERT	O
LARGE	O
outperforms	O
the	O
ranker	O
by	O
almost	O
6	O
percentage	O
points	O
in	O
F	O
1	O
macro	O
.	O

We	O
aim	O
to	O
train	O
a	O
model	O
that	O
is	O
able	O
to	O
transfer	O
argumentative	O
information	O
concisely	O
within	O
a	O
single	O
sentence	O
.	O
We	O
define	O
such	O
an	O
argument	O
as	O
the	O
combination	O
of	O
a	O
topic	O
and	O
a	O
sentence	O
holding	O
evidence	O
with	O
a	O
specific	O
stance	O
towards	O
this	O
topic	O
(	O
Stab	O
et	O
al	O
.	O
,	O
2018b	O
)	O
.	O
Consequently	O
,	O
the	O
following	O
preprocessing	O
steps	O
ultimately	O
target	O
retrieval	O
and	O
classification	O
of	O
sentences	O
.	O
To	O
evaluate	O
different	O
data	O
sources	O
,	O
we	O
use	O
a	O
dump	O
from	O
Common-	O
We	O
notice	O
that	O
many	O
sentences	O
are	O
not	O
relevant	O
with	O
regard	O
to	O
the	O
document	O
's	O
topic	O
.	O
To	O
enforce	O
topicrelevance	O
,	O
we	O
decide	O
to	O
filter	O
out	O
all	O
sentences	O
that	O
do	O
not	O
contain	O
at	O
least	O
one	O
token	O
of	O
the	O
respective	O
topic	O
or	O
its	O
defined	O
synonyms	O
(	O
see	O
App	O
.	O
B	O
)	O
.	O
We	O
use	O
the	O
ArgumenText	O
API	O
's	O
6	O
argument	O
and	O
stance	O
classification	O
models	O
(	O
Stab	O
et	O
al	O
.	O
,	O
2018a	O
)	O
to	O
classify	O
all	O
sentences	O
into	O
argument	O
or	O
non	O
-	O
argument	O
(	O
F	O
1	O
macro	O
=	O
.7384	O
)	O
,	O
and	O
remaining	O
arguments	O
into	O
pro	O
or	O
con	O
with	O
regard	O
to	O
the	O
topic	O
(	O
F	O
1	O
macro	O
=	O
.7661	O
)	O
.	O

Aspect	O
Detection	O
We	O
detect	O
aspects	O
on	O
all	O
remaining	O
arguments	O
.	O
To	O
speed	O
up	O
the	O
detection	O
on	O
millions	O
of	O
sentences	O
,	O
we	O
use	O
BERT	O
BASE	O
instead	O
of	O
BERT	O
LARGE	O
(	O
see	O
Table	O
3	O
)	O
.	O

Training	O
Document	O
Generation	O
We	O
create	O
the	O
final	O
training	O
documents	O
for	O
the	O
argument	O
generation	O
model	O
by	O
concatenating	O
all	O
arguments	O
that	O
have	O
the	O
same	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
(	O
i.e.	O
the	O
same	O
control	O
code	O
)	O
.	O
Further	O
,	O
we	O
aggregate	O
all	O
arguments	O
that	O
include	O
an	O
aspect	O
with	O
the	O
same	O
stem	O
into	O
the	O
same	O
document	O
(	O
e.g.	O
arguments	O
with	O
cost	O
and	O
costs	O
as	O
aspect	O
)	O
.	O
To	O
cope	O
with	O
limited	O
hardware	O
resources	O
,	O
we	O
restrict	O
the	O
total	O
number	O
of	O
arguments	O
for	O
each	O
topic	O
and	O
stance	O
to	O
100,000	O
(	O
i.e.	O
1.6	O
M	O
over	O
all	O
eight	O
topics	O
)	O
.	O
Also	O
,	O
as	O
some	O
aspects	O
dominate	O
by	O
means	O
of	O
quantity	O
of	O
related	O
arguments	O
and	O
others	O
appear	O
only	O
rarely	O
,	O
we	O
empirically	O
determine	O
an	O
upper	O
and	O
lower	O
bound	O
of	O
1,500	O
and	O
15	O
arguments	O
for	O
each	O
document	O
,	O
which	O
still	O
allows	O
us	O
to	O
retrieve	O
the	O
above	O
defined	O
amount	O
of	O
training	O
arguments	O
.	O

Model	O
Training	O
and	O
Analysis	O

In	O
the	O
following	O
,	O
we	O
describe	O
the	O
architecture	O
and	O
the	O
training	O
process	O
of	O
the	O
Arg	O
-	O
CTRL	O
and	O
analyze	O
its	O
performance	O
.	O

Model	O
and	O
Training	O

Model	O
The	O
goal	O
of	O
a	O
statistical	O
language	O
model	O
is	O
to	O
learn	O
the	O
conditional	O
probability	O
of	O
the	O
next	O
word	O
given	O
all	O
(	O
or	O
a	O
subset	O
of	O
)	O
the	O
previous	O
ones	O
(	O
Bengio	O
et	O
al	O
.	O
,	O
2003	O
)	O
.	O
That	O
is	O
,	O
for	O
a	O
sequence	O
of	O
tokens	O
x	O
=	O
(	O
x	O
1	O
,	O
...	O
,	O
x	O
n	O
)	O
,	O
the	O
model	O
learns	O
p(x	O
i	O
|x	O
<	O
i	O
)	O
where	O
x	O
i	O
is	O
the	O
i	O
-	O
th	O
word	O
of	O
sequence	O
x.	O
For	O
this	O
work	O
,	O
we	O
use	O
the	O
1.63	O
billion	O
-	O
parameter	O
Conditional	O
Transformer	O
Language	O
Model	O
(	O
CTRL	O
)	O
by	O
Keskar	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
which	O
is	O
built	O
on	O
a	O
transformerbased	O
sequence	O
to	O
sequence	O
architecture	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
The	O
CTRL	O
has	O
shown	O
to	O
produce	O
high	O
quality	O
text	O
,	O
is	O
general	O
enough	O
to	O
be	O
adapted	O
for	O
conditioning	O
on	O
the	O
control	O
codes	O
we	O
aim	O
to	O
use	O
,	O
and	O
we	O
do	O
not	O
need	O
to	O
pre	O
-	O
train	O
the	O
weights	O
from	O
scratch	O
.	O
Formally	O
,	O
the	O
CTRL	O
adds	O
an	O
extra	O
condition	O
to	O
each	O
sequence	O
by	O
prepending	O
a	O
control	O
code	O
c	O
,	O
hence	O
learning	O
p(x	O
i	O
|x	O
<	O
i	O
,	O
c	O
)	O
.	O
The	O
control	O
code	O
is	O
represented	O
by	O
a	O
single	O
token	O
and	O
can	O
then	O
be	O
used	O
to	O
direct	O
the	O
model	O
output	O
at	O
inference	O
.	O
We	O
extend	O
the	O
model	O
from	O
its	O
previous	O
limit	O
of	O
a	O
singletoken	O
control	O
code	O
to	O
accept	O
multiple	O
tokens	O
.	O
For	O
The	O
respective	O
control	O
code	O
is	O
prepended	O
to	O
each	O
sequence	O
of	O
256	O
subwords	O
of	O
a	O
document	O
.	O

Analysis	O

Generation	O
At	O
inference	O
,	O
we	O
gather	O
multiple	O
generated	O
arguments	O
from	O
a	O
control	O
code	O
input	O
by	O
splitting	O
the	O
generated	O
output	O
text	O
into	O
sentences	O
with	O
NLTK	O
(	O
Bird	O
et	O
al	O
.	O
,	O
2009	O
)	O
.	O
We	O
observe	O
that	O
for	O
the	O
first	O
generated	O
argument	O
,	O
the	O
Arg	O
-	O
CTRL	O
mostly	O
outputs	O
very	O
short	O
phrases	O
,	O
as	O
it	O
tries	O
to	O
incorporate	O
the	O
control	O
code	O
into	O
a	O
meaningful	O
start	O
of	O
an	O
argument	O
.	O
We	O
prevent	O
this	O
by	O
adding	O
punctuation	O
marks	O
after	O
each	O
control	O
code	O
(	O
e.g.	O
a	O
period	O
or	O
colon	O
)	O
,	O
signaling	O
the	O
model	O
to	O
start	O
a	O
new	O
sentence	O
.	O
In	O
this	O
fashion	O
,	O
we	O
generate	O
proand	O
con	O
-	O
arguments	O
up	O
to	O
the	O
pre	O
-	O
defined	O
training	O
split	O
size	O
7	O
for	O
each	O
topic	O
of	O
the	O
UKP	O
-	O
Corpus	O
,	O
resulting	O
in	O
7,991	O
newly	O
generated	O
arguments	O
.	O
We	O
do	O
this	O
with	O
both	O
models	O
and	O
use	O
the	O
generated	O
arguments	O
as	O
a	O
basis	O
for	O
the	O
following	O
analysis	O
and	O
evaluation	O
methods	O
.	O
Examples	O
of	O
generated	O
arguments	O
can	O
be	O
found	O
in	O
Tables	O
4	O
,	O
6	O
,	O
and	O
7	O
(	O
as	O
part	O
of	O
the	O
evaluation	O
,	O
see	O
Section	O
7	O
)	O
.	O

Results	O
With	O
no	O
other	O
previous	O
work	O
on	O
explicit	O
control	O
of	O
argument	O
generation	O
(	O
to	O
the	O
best	O
of	O
our	O
knowledge	O
)	O
,	O
we	O
decide	O
to	O
proof	O
our	O
concept	O
of	O
aspect	O
-	O
controlled	O
neural	O
argument	O
generation	O
by	O
7	O
Not	O
counting	O
non	O
-	O
arguments	O
from	O
the	O
splits	O
.	O

comparing	O
both	O
generation	O
models	O
to	O
a	O
retrieval	O
approach	O
as	O
a	O
strong	O
upper	O
bound	O
.	O
The	O
retrieval	O
approach	O
returns	O
all	O
arguments	O
from	O
the	O
classified	O
training	O
data	O
(	O
see	O
Section	O
4	O
)	O
that	O
match	O
a	O
given	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
.	O
Both	O
the	O
retrieval	O
and	O
generation	O
approaches	O
are	O
evaluated	O
against	O
reference	O
data	O
from	O
debate	O
portals	O
and	O
compared	O
via	O
METEOR	O
(	O
Lavie	O
and	O
Agarwal	O
,	O
2007	O
)	O
and	O
ROUGE	O
-	O
L	O
(	O
Lin	O
,	O
2004	O
)	O
metrics	O
.	O
The	O
retrieval	O
approach	O
has	O
an	O
advantage	O
in	O
this	O
setup	O
,	O
as	O
the	O
arguments	O
are	O
also	O
of	O
human	O
origin	O
and	O
aspects	O
are	O
always	O
explicitly	O
stated	O
within	O
a	O
belonging	O
argument	O
.	O

The	O
reference	O
data	O
was	O
crawled	O
from	O
two	O
debate	O
portals	O
8	O
and	O
consists	O
of	O
pro	O
-	O
and	O
con	O
-	O
paragraphs	O
discussing	O
the	O
eight	O
topics	O
of	O
the	O
UKP	O
-	O
Corpus	O
.	O
As	O
the	O
paragraphs	O
may	O
include	O
non	O
-	O
arguments	O
,	O
we	O
filter	O
these	O
out	O
by	O
classifying	O
all	O
sentences	O
with	O
the	O
ArgumenText	O
API	O
into	O
arguments	O
and	O
nonarguments	O
.	O
This	O
leaves	O
us	O
with	O
349	O
pro	O
-	O
and	O
355	O
con	O
-	O
arguments	O
over	O
all	O
topics	O
(	O
see	O
App	O
.	O
D	O
for	O
the	O
topic	O
-	O
wise	O
distribution	O
)	O
.	O
Next	O
,	O
we	O
detect	O
all	O
aspects	O
in	O
these	O
arguments	O
.	O
Arguments	O
with	O
the	O
same	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
are	O
then	O
grouped	O
and	O
used	O
as	O
reference	O
for	O
arguments	O
from	O
the	O
(	O
a	O
)	O
generated	O
arguments	O
and	O
(	O
b	O
)	O
retrieval	O
approach	O
arguments	O
if	O
these	O
hold	O
the	O
same	O
topic	O
,	O
stance	O
,	O
and	O
aspect	O
.	O
The	O
results	O
reveal	O
that	O
both	O
the	O
average	O
METEOR	O
and	O
ROUGE	O
-	O
L	O
scores	O
are	O
only	O
marginally	O
lower	O
than	O
the	O
retrieval	O
scores	O
(	O
METEOR	O
is	O
0.5/1.1	O
points	O
lower	O
for	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
/Arg	O
-	O
CTRL	O
CC	O
,	O
see	O
Table	O
5	O
)	O
.	O
It	O
not	O
only	O
shows	O
the	O
strength	O
of	O
the	O
architecture	O
,	O
but	O
also	O
the	O
success	O
in	O
generating	O
sound	O
aspect	O
-	O
specific	O
arguments	O
with	O
our	O
approach	O
.	O
Overlap	O
with	O
Training	O
Data	O
We	O
find	O
arguments	O
generated	O
by	O
the	O
models	O
to	O
be	O
genuine	O
,	O
i.e.	O
demonstrating	O
substantial	O
differences	O
to	O
the	O
training	O
data	O
.	O
For	O
each	O
of	O
the	O
7,991	O
generated	O
arguments	O
,	O
we	O
find	O
the	O
most	O
similar	O
argument	O
in	O
the	O
training	O
data	O
based	O
on	O
the	O
cosine	O
similarity	O
of	O
their	O
BERT	O
embeddings	O
We	O
compare	O
all	O
models	O
by	O
verifying	O
whether	O
or	O
not	O
the	O
aspect	O
used	O
for	O
generation	O
(	O
including	O
synonyms	O
and	O
their	O
stems	O
and	O
lemmas	O
)	O
can	O
be	O
found	O
in	O
the	O
generated	O
arguments	O
.	O
For	O
the	O
original	O
models	O
conditioned	O
on	O
aspects	O
,	O
this	O
is	O
true	O
in	O
79	O
%	O
of	O
Generated	O
sentence	O
:	O
We	O
do	O
n't	O
need	O
more	O
gun	O
control	O
laws	O
when	O
we	O
already	O
have	O
enough	O
restrictions	O
on	O
who	O
can	O
buy	O
guns	O
in	O
this	O
country	O
.	O

Training	O
sentence	O
:	O
We	O
have	O
some	O
of	O
the	O
strongest	O
gun	O
laws	O
in	O
the	O
country	O
,	O
but	O
guns	O
do	O
n't	O
respect	O
boundaries	O
any	O
more	O
than	O
criminals	O
do	O
.	O
Cosine	O
similarity	O
/	O
edit	O
distance	O
/	O
rel	O
.	O
overlap	O
:	O
95.59	O
/	O
88	O
/	O
8	O
%	O
Generated	O
sentence	O
:	O
The	O
radioactivity	O
of	O
the	O
spent	O
fuel	O
is	O
a	O
concern	O
,	O
as	O
it	O
can	O
be	O
used	O
to	O
make	O
weapons	O
and	O
has	O
been	O
linked	O
to	O
cancer	O
in	O
humans	O
.	O
Training	O
sentence	O
:	O
However	O
,	O
it	O
does	O
produce	O
radioactive	O
waste	O
,	O
which	O
must	O
be	O
disposed	O
of	O
carefully	O
as	O
it	O
can	O
cause	O
health	O
problems	O
and	O
can	O
be	O
used	O
to	O
make	O
nuclear	O
weapons	O
Cosine	O
similarity	O
/	O
edit	O
distance	O
/	O
rel	O
.	O
overlap	O
:	O
92.40	O
/	O
99	O
/	O
17	O
%	O
the	O
cases	O
for	O
Arg	O
-	O
CTRL	O
REDDIT	O
and	O
in	O
74	O
%	O
of	O
the	O
cases	O
for	O
Arg	O
-	O
CTRL	O
CC	O
.	O
For	O
the	O
model	O
that	O
was	O
not	O
conditioned	O
on	O
aspects	O
,	O
however	O
,	O
it	O
is	O
only	O
true	O
in	O
8	O
%	O
of	O
the	O
cases	O
.	O
It	O
clearly	O
shows	O
the	O
necessity	O
to	O
condition	O
the	O
model	O
on	O
aspects	O
explicitly	O
,	O
implying	O
the	O
need	O
for	O
argument	O
aspect	O
detection	O
,	O
as	O
the	O
model	O
is	O
unable	O
to	O
learn	O
generating	O
aspect	O
-	O
related	O
arguments	O
otherwise	O
.	O
Moreover	O
,	O
without	O
prior	O
detection	O
of	O
aspects	O
,	O
we	O
have	O
no	O
means	O
for	O
proper	O
aggregation	O
over	O
aspects	O
.	O
We	O
notice	O
that	O
for	O
the	O
model	O
without	O
prior	O
knowledge	O
of	O
aspects	O
,	O
79	O
%	O
of	O
all	O
aspects	O
in	O
the	O
training	O
data	O
appear	O
in	O
only	O
one	O
argument	O
.	O
For	O
these	O
aspects	O
,	O
the	O
model	O
will	O
likely	O
not	O
pick	O
up	O
a	O
strong	O
enough	O
signal	O
to	O
learn	O
them	O
.	O

Evaluation	O

We	O
evaluate	O
the	O
quality	O
(	O
intrinsic	O
evaluation	O
)	O
of	O
the	O
Arg	O
-	O
CTRL	O
and	O
its	O
performance	O
on	O
an	O
exemplary	O
task	O
(	O
extrinsic	O
evaluation	O
)	O
.	O
As	O
a	O
basis	O
,	O
we	O
use	O
the	O
7,991	O
arguments	O
generated	O
in	O
Section	O
5	O
.	O

Intrinsic	O
Evaluation	O

Human	O
Evaluation	O
We	O
conduct	O
an	O
expert	O
evaluation	O
on	O
a	O
subset	O
of	O
generated	O
arguments	O
with	O
two	O
researchers	O
(	O
field	O
of	O
expertise	O
is	O
natural	O
language	O
processing	O
)	O
not	O
involved	O
in	O
this	O
paper	O
.	O
Two	O
aspects	O
are	O
evaluated	O
:	O
fluency	O
and	O
persuasiveness	O
.	O
We	O
consider	O
a	O
sentence	O
as	O
fluent	O
if	O
it	O
is	O
grammatically	O
correct	O
,	O
i.e.	O
contains	O
neither	O
semantic	O
nor	O
syntactic	O
errors	O
,	O
and	O
arrange	O
this	O
as	O
a	O
binary	O
task	O
.	O
To	O
reduce	O
subjectivity	O
for	O
the	O
persuasiveness	O
evaluation	O
,	O
the	O
experts	O
do	O
not	O
annotate	O
single	O
arguments	O
but	O
instead	O
compare	O
pairs	O
(	O
Habernal	O
and	O
Gurevych	O
,	O
2016	O
)	O
of	O
generated	O
and	O
refer	O
-	O
ence	O
data	O
arguments	O
(	O
see	O
Section	O
5.2	O
)	O
.	O
The	O
experts	O
could	O
either	O
choose	O
one	O
argument	O
as	O
being	O
more	O
persuasive	O
or	O
both	O
as	O
being	O
equally	O
persuasive	O
.	O
In	O
total	O
,	O
the	O
experts	O
compared	O
100	O
(	O
randomly	O
sorted	O
and	O
ordered	O
)	O
argument	O
pairs	O
for	O
persuasiveness	O
and	O
fluency	O
(	O
50	O
from	O
both	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
and	O
the	O
Arg	O
-	O
CTRL	O
CC	O
)	O
.	O
A	O
pair	O
of	O
arguments	O
always	O
had	O
the	O
same	O
topic	O
and	O
stance	O
.	O
For	O
fluency	O
,	O
only	O
the	O
annotations	O
made	O
for	O
generated	O
arguments	O
were	O
extracted	O
and	O
taken	O
into	O
account	O
.	O
Averaged	O
results	O
of	O
both	O
experts	O
show	O
that	O
in	O
33	O
%	O
of	O
the	O
cases	O
,	O
the	O
generated	O
argument	O
is	O
either	O
more	O
convincing	O
(	O
29	O
%	O
)	O
or	O
as	O
convincing	O
(	O
4	O
%	O
)	O
as	O
the	O
reference	O
argument	O
.	O
Moreover	O
,	O
83	O
%	O
of	O
generated	O
arguments	O
are	O
fluent	O
.	O
The	O
inter	O
-	O
annotator	O
agreement	O
(	O
Cohen	O
,	O
1960	O
)	O
between	O
the	O
two	O
experts	O
is	O
Cohen	O
's	O
κ	O
=	O
.30	O
(	O
percentage	O
agreement	O
:	O
.62	O
)	O
for	O
persuasiveness	O
and	O
κ	O
=	O
.43	O
(	O
percentage	O
agreement	O
:	O
.72	O
)	O
for	O
fluency	O
,	O
which	O
can	O
be	O
interpreted	O
as	O
"	O
fair	O
"	O
and	O
"	O
moderate	O
"	O
agreement	O
,	O
respectively	O
(	O
Landis	O
and	O
Koch	O
,	O
1977	O
)	O
.	O
As	O
we	O
compare	O
to	O
high	O
-	O
quality	O
,	O
curated	O
data	O
,	O
the	O
perceived	O
persuasiveness	O
of	O
the	O
generated	O
arguments	O
shows	O
the	O
potential	O
of	O
the	O
work	O
-	O
further	O
strengthened	O
in	O
the	O
remainder	O
of	O
this	O
section	O
.	O

Argument	O
Quality	O
We	O
introduce	O
a	O
novel	O
method	O
to	O
evaluate	O
generated	O
arguments	O
based	O
on	O
the	O
argument	O
quality	O
detection	O
approach	O
proposed	O
by	O
Gretz	O
et	O
al	O
.	O
(	O
2020b	O
)	O
.	O
They	O
create	O
an	O
argument	O
quality	O
dataset	O
that	O
contains	O
around	O
30,000	O
arguments	O
over	O
71	O
topics	O
.	O
For	O
each	O
argument	O
,	O
annotators	O
were	O
asked	O
whether	O
or	O
not	O
they	O
would	O
recommend	O
a	O
friend	O
to	O
use	O
the	O
displayed	O
argument	O
in	O
a	O
speech	O
.	O
The	O
quality	O
scores	O
for	O
each	O
argument	O
result	O
from	O
a	O
weighted	O
average	O
(	O
WA	O
)	O
or	O
MACE	O
Probability	O
function	O
of	O
all	O
annotations	O
and	O
range	O
between	O
0	O
(	O
lowest	O
quality	O
)	O
and	O
1.0	O
(	O
highest	O
quality	O
)	O
.	O
We	O
use	O
the	O
WA	O
-	O
score	O
as	O
label	O
,	O
the	O
same	O
model	O
(	O
BERT	O
BASE	O
)	O
and	O
hyperparameters	O
as	O
given	O
in	O
the	O
original	O
paper	O
,	O
and	O
reproduce	O
the	O
reported	O
correlations	O
of	O
.52	O
(	O
Pearson	O
)	O
and	O
.48	O
(	O
Spearman	O
)	O
on	O
the	O
test	O
dataset	O
(	O
averaged	O
over	O
five	O
different	O
seeds	O
)	O
.	O
The	O
model	O
predicts	O
an	O
average	O
argument	O
quality	O
of	O
.71	O
for	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
arguments	O
,	O
.75	O
for	O
the	O
training	O
arguments	O
of	O
the	O
UKP	O
-	O
Corpus	O
,	O
and	O
even	O
.76	O
for	O
the	O
Arg	O
-	O
CTRL	O
CC	O
arguments	O
.	O
It	O
shows	O
that	O
our	O
model	O
is	O
able	O
to	O
produce	O
arguments	O
that	O
are	O
generally	O
on	O
a	O
similar	O
quality	O
level	O
as	O
arguments	O
made	O
by	O
humans	O
.	O
The	O
difference	O
of	O
5	O
points	O
between	O
the	O
Arg	O
-	O
CTRL	O
CC	O
and	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
,	O
however	O
,	O
shows	O
that	O
training	O
on	O
REDDIT	O
data	O
leads	O
to	O
a	O
lower	O
quality	O
of	O
generated	O
arguments	O
.	O
Exemplary	O
Human	O
:	O
Nuclear	O
energy	O
produces	O
waste	O
that	O
stays	O
radioactive	O
for	O
thousands	O
of	O
years	O
and	O
pollutes	O
the	O
environment	O
.	O
Arg	O
-	O
CTRLCC	O
:	O
(	O
nuclear	O
energy	O
PRO	O
waste	O
.	O
)	O
The	O
only	O
thing	O
that	O
's	O
going	O
to	O
be	O
left	O
over	O
is	O
the	O
spent	O
fuel	O
rods	O
,	O
which	O
are	O
not	O
very	O
radioactive	O
and	O
can	O
be	O
reprocessed	O
[	O
...	O
]	O
Arg	O
-	O
CTRLCC	O
:	O
(	O
nuclear	O
energy	O
PRO	O
radioactive	O
.	O
)	O
The	O
radioactivity	O
of	O
the	O
waste	O
is	O
not	O
a	O
problem	O
,	O
because	O
it	O
will	O
decay	O
away	O
in	O
about	O
300	O
years	O
or	O
so	O
[	O
...	O
]	O
Arg	O
-	O
CTRLCC	O
:	O
(	O
nuclear	O
energy	O
PRO	O
environment	O
.	O
)	O
The	O
fact	O
that	O
the	O
radioactive	O
waste	O
is	O
not	O
only	O
stored	O
in	O
a	O
safe	O
manner	O
,	O
but	O
also	O
protected	O
from	O
any	O
potential	O
threat	O
to	O
human	O
health	O
or	O
the	O
environment	O
for	O
thousands	O
of	O
years	O
[	O
...	O
]	O
Human	O
:	O
School	O
uniforms	O
are	O
expensive	O
and	O
affect	O
the	O
pupil	O
's	O
individuality	O
.	O
Arg	O
-	O
CTRLCC	O
:	O
(	O
school	O
uniforms	O
PRO	O
expensive	O
.	O
)	O
The	O
cost	O
of	O
the	O
uniform	O
is	O
not	O
that	O
much	O
more	O
than	O
a	O
regular	O
shirt	O
and	O
shorts	O
[	O
...	O
]	O
Arg	O
-	O
CTRLCC	O
:	O
(	O
school	O
uniforms	O
PRO	O
individuality	O
.	O
)	O
The	O
uniform	O
is	O
a	O
symbol	O
of	O
unity	O
and	O
identity	O
,	O
which	O
helps	O
to	O
foster	O
an	O
environment	O
where	O
students	O
can	O
feel	O
comfortable	O
about	O
expressing	O
their	O
own	O
individual	O
style	O
without	O
being	O
judged	O
[	O
...	O
]	O
for	O
three	O
topics	O
,	O
we	O
show	O
the	O
generated	O
arguments	O
with	O
the	O
highest	O
and	O
lowest	O
argument	O
quality	O
in	O
Table	O
4	O
(	O
see	O
App	O
.	O
E	O
for	O
the	O
full	O
table	O
)	O
.	O

Extrinsic	O
Evaluation	O
:	O

Counter	O
-	O
Arguments	O

Drafting	O
counter	O
-	O
arguments	O
is	O
an	O
important	O
skill	O
for	O
debating	O
,	O
to	O
provide	O
constructive	O
feedback	O
,	O
and	O
to	O
foster	O
critical	O
thinking	O
.	O
We	O
lean	O
onto	O
the	O
work	O
of	O
Wachsmuth	O
et	O
al	O
.	O
(	O
2018	O
)	O
who	O
describe	O
a	O
counterargument	O
as	O
discussing	O
the	O
same	O
aspect	O
as	O
an	O
initial	O
argument	O
,	O
but	O
with	O
a	O
switched	O
stance	O
.	O
Hence	O
,	O
given	O
our	O
defined	O
control	O
codes	O
,	O
our	O
model	O
is	O
especially	O
fit	O
for	O
counter	O
-	O
argument	O
generation	O
.	O
Unlike	O
current	O
models	O
for	O
this	O
task	O
,	O
we	O
do	O
not	O
require	O
a	O
specific	O
dataset	O
with	O
argument	O
and	O
counterargument	O
pairs	O
(	O
Hidey	O
and	O
McKeown	O
,	O
2019	O
;	O
.	O
Also	O
,	O
in	O
contrast	O
to	O
the	O
model	O
by	O
that	O
implicitly	O
integrates	O
inputrelated	O
"	O
Keyphrases	O
"	O
into	O
the	O
process	O
of	O
counterargument	O
generation	O
,	O
our	O
model	O
is	O
able	O
to	O
concentrate	O
on	O
every	O
aspect	O
of	O
the	O
input	O
explicitly	O
and	O
with	O
a	O
separate	O
argument	O
,	O
allowing	O
for	O
more	O
transparency	O
and	O
interpretability	O
over	O
the	O
process	O
of	O
counter	O
-	O
argument	O
generation	O
.	O
We	O
exemplary	O
show	O
how	O
the	O
combination	O
of	O
aspect	O
detection	O
and	O
controlled	O
argument	O
generation	O
can	O
be	O
successfully	O
leveraged	O
to	O
tackle	O
this	O
task	O
.	O
For	O
that	O
,	O
we	O
manually	O
compose	O
initial	O
arguments	O
for	O
the	O
topics	O
nuclear	O
energy	O
and	O
school	O
uniforms	O
.	O
Then	O
,	O
we	O
automatically	O
detect	O
their	O
aspects	O
and	O
generate	O
a	O
counterargument	O
for	O
each	O
aspect	O
by	O
passing	O
the	O
topic	O
,	O
opposite	O
stance	O
of	O
the	O
original	O
argument	O
,	O
and	O
one	O
of	O
the	O
aspects	O
into	O
the	O
Arg	O
-	O
CTRL	O
CC	O
.	O
For	O
both	O
topics	O
,	O
the	O
Arg	O
-	O
CTRL	O
CC	O
produces	O
meaningful	O
counterarguments	O
based	O
on	O
the	O
detected	O
aspects	O
(	O
see	O
Table	O
7	O
)	O
.	O

Conclusion	O

We	O
apply	O
the	O
concept	O
of	O
controlled	O
neural	O
text	O
generation	O
to	O
the	O
domain	O
of	O
argument	O
generation	O
.	O
Our	O
Arg	O
-	O
CTRL	O
is	O
conditioned	O
on	O
topics	O
,	O
stances	O
,	O
and	O
aspects	O
and	O
can	O
reliably	O
create	O
arguments	O
using	O
these	O
control	O
codes	O
.	O
We	O
show	O
that	O
arguments	O
generated	O
with	O
our	O
approach	O
are	O
genuine	O
and	O
of	O
high	O
argumentative	O
and	O
grammatical	O
quality	O
in	O
general	O
.	O
Moreover	O
,	O
we	O
show	O
that	O
our	O
approach	O
can	O
be	O
used	O
to	O
generate	O
counter	O
-	O
arguments	O
in	O
a	O
transparent	O
and	O
interpretable	O
way	O
.	O
We	O
fine	O
-	O
tune	O
the	O
Arg	O
-	O
CTRL	O
on	O
two	O
different	O
data	O
sources	O
and	O
find	O
that	O
using	O
mixed	O
data	O
from	O
Common	O
-	O
Crawl	O
results	O
in	O
a	O
higher	O
quality	O
of	O
generated	O
arguments	O
than	O
using	O
user	O
discussions	O
from	O
Reddit	O
-	O
Comments	O
.	O
Further	O
,	O
we	O
define	O
argument	O
aspect	O
detection	O
for	O
controlled	O
argument	O
generation	O
and	O
introduce	O
a	O
novel	O
annotation	O
scheme	O
to	O
crowdsource	O
argument	O
aspect	O
annotations	O
,	O
resulting	O
in	O
a	O
high	O
-	O
quality	O
dataset	O
.	O
We	O
publish	O
the	O
model	O
weights	O
,	O
data	O
,	O
and	O
all	O
code	O
necessary	O
to	O
train	O
the	O
Arg	O
-	O
CTRL	O
.	O

Ethics	O
Statement	O

Models	O
for	O
argument	O
and	O
claim	O
generation	O
have	O
been	O
discussed	O
in	O
our	O
related	O
work	O
and	O
are	O
widely	O
available	O
.	O
Gretz	O
et	O
al	O
.	O
(	O
2020a	O
)	O
suggest	O
that	O
,	O
in	O
order	O
to	O
allow	O
for	O
a	O
fine	O
-	O
grained	O
control	O
over	O
claim	O
/	O
argument	O
generation	O
,	O
aspect	O
selection	O
needs	O
to	O
be	O
handled	O
carefully	O
,	O
which	O
is	O
what	O
we	O
have	O
focused	O
on	O
in	O
this	O
work	O
.	O
The	O
dangers	O
of	O
misuse	O
of	O
language	O
models	O
like	O
the	O
CTRL	O
have	O
been	O
extensively	O
discussed	O
by	O
its	O
authors	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
The	O
ethical	O
impact	O
of	O
these	O
works	O
has	O
been	O
weighed	O
and	O
deemed	O
justifiable	O
.	O
Argument	O
generation	O
-	O
and	O
natural	O
language	O
generation	O
as	O
a	O
whole	O
-	O
is	O
subject	O
to	O
dual	O
use	O
.	O
The	O
technology	O
can	O
be	O
used	O
to	O
create	O
arguments	O
that	O
can	O
not	O
be	O
distinguished	O
from	O
human	O
-	O
made	O
arguments	O
.	O
While	O
our	O
intentions	O
are	O
to	O
support	O
society	O
,	O
to	O
foster	O
diversity	O
in	O
debates	O
,	O
and	O
to	O
encourage	O
research	O
on	O
this	O
important	O
topic	O
,	O
we	O
are	O
aware	O
of	O
the	O
possibility	O
of	O
harmful	O
applications	O
this	O
model	O
can	O
be	O
used	O
for	O
.	O
For	O
instance	O
,	O
the	O
model	O
could	O
be	O
used	O
to	O
generate	O
only	O
opposing	O
(	O
or	O
supporting	O
)	O
arguments	O
on	O
one	O
of	O
the	O
pretrained	O
topics	O
and	O
aspects	O
and	O
,	O
as	O
such	O
,	O
bias	O
a	O
debate	O
into	O
a	O
certain	O
direction	O
.	O
Also	O
,	O
bots	O
could	O
use	O
the	O
generated	O
arguments	O
to	O
spread	O
them	O
via	O
social	O
media	O
.	O
The	O
same	O
is	O
true	O
,	O
however	O
,	O
for	O
argument	O
search	O
engines	O
,	O
which	O
can	O
be	O
used	O
by	O
malicious	O
parties	O
to	O
retrieve	O
(	O
and	O
then	O
spread	O
)	O
potentially	O
harmful	O
information	O
.	O

However	O
,	O
controllable	O
argument	O
generation	O
can	O
also	O
be	O
used	O
to	O
support	O
finding	O
and	O
formulating	O
(	O
counter-)arguments	O
for	O
debates	O
,	O
for	O
writing	O
essays	O
,	O
to	O
enrich	O
one	O
-	O
sided	O
discussions	O
,	O
and	O
thus	O
,	O
to	O
make	O
discourse	O
more	O
diverse	O
overall	O
.	O
For	O
instance	O
,	O
anticipating	O
opposing	O
arguments	O
is	O
crucial	O
for	O
critical	O
thinking	O
,	O
which	O
is	O
the	O
foundation	O
for	O
any	O
democratic	O
society	O
.	O
The	O
skill	O
is	O
extensively	O
taught	O
in	O
school	O
and	O
university	O
education	O
.	O
However	O
,	O
confirmation	O
bias	O
(	O
or	O
myside	O
bias	O
)	O
(	O
Stanovich	O
et	O
al	O
.	O
,	O
2013	O
)	O
,	O
i.e.	O
the	O
tendency	O
to	O
ignore	O
opposing	O
arguments	O
,	O
is	O
an	O
ever	O
-	O
present	O
issue	O
.	O
Technologies	O
like	O
ours	O
could	O
be	O
used	O
to	O
mitigate	O
this	O
issue	O
by	O
,	O
for	O
instance	O
,	O
automatically	O
providing	O
topic	O
-	O
and	O
aspectspecific	O
counter	O
-	O
arguments	O
for	O
all	O
arguments	O
of	O
a	O
given	O
text	O
(	O
this	O
has	O
been	O
shown	O
for	O
single	O
arguments	O
in	O
Section	O
7.2	O
)	O
.	O
We	O
believe	O
that	O
working	O
on	O
and	O
providing	O
access	O
to	O
such	O
models	O
is	O
of	O
major	O
importance	O
and	O
,	O
overall	O
,	O
a	O
benefit	O
to	O
society	O
.	O

Open	O
-	O
sourcing	O
such	O
language	O
models	O
also	O
encourages	O
the	O
work	O
on	O
counter	O
-	O
measures	O
to	O
detect	O
malicious	O
use	O
:	O
While	O
many	O
works	O
have	O
been	O
published	O
on	O
the	O
topic	O
of	O
automatic	O
fake	O
news	O
detection	O
in	O
texts	O
(	O
Kaliyar	O
et	O
al	O
.	O
,	O
2020;Reis	O
et	O
al	O
.	O
,	O
2019;Hanselowski	O
et	O
al	O
.	O
,	O
2018;Pérez	O
-	O
Rosas	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
the	O
recent	O
emergence	O
of	O
large	O
-	O
scale	O
language	O
models	O
has	O
also	O
encouraged	O
research	O
to	O
focus	O
on	O
detecting	O
the	O
creator	O
of	O
these	O
texts	O
(	O
Varshney	O
et	O
al	O
.	O
,	O
2020;Zellers	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
The	O
former	O
approaches	O
are	O
aimed	O
at	O
detecting	O
fake	O
news	O
in	O
general	O
,	O
i.e.	O
independent	O
of	O
who	O
(	O
or	O
what	O
)	O
composed	O
a	O
text	O
,	O
whereas	O
the	O
latter	O
approaches	O
are	O
designed	O
to	O
recognize	O
if	O
a	O
text	O
was	O
written	O
by	O
a	O
human	O
or	O
generated	O
by	O
a	O
language	O
model	O
.	O
We	O
encourage	O
the	O
work	O
on	O
both	O
types	O
of	O
methods	O
.	O
Ideally	O
,	O
social	O
networks	O
and	O
news	O
platforms	O
would	O
indicate	O
if	O
a	O
statement	O
was	O
automatically	O
generated	O
in	O
addition	O
to	O
its	O
factual	O
correctness	O
.	O

Further	O
,	O
we	O
point	O
out	O
some	O
limitations	O
of	O
the	O
Arg	O
-	O
CTRL	O
that	O
mitigate	O
the	O
risks	O
discussed	O
before	O
.	O
One	O
of	O
these	O
limitations	O
is	O
that	O
it	O
can	O
not	O
be	O
used	O
to	O
generate	O
arguments	O
for	O
unseen	O
topics	O
,	O
which	O
makes	O
a	O
widespread	O
application	O
(	O
e.g.	O
to	O
produce	O
fake	O
news	O
)	O
rather	O
unlikely	O
(	O
using	O
an	O
unseen	O
topic	O
as	O
control	O
code	O
results	O
in	O
nonsensical	O
repetitions	O
of	O
the	O
input	O
)	O
.	O
The	O
analysis	O
in	O
Section	O
6	O
of	O
the	O
paper	O
shows	O
that	O
the	O
model	O
fails	O
to	O
produce	O
aspectspecific	O
sentences	O
in	O
92	O
%	O
of	O
the	O
cases	O
if	O
it	O
was	O
not	O
explicitly	O
conditioned	O
on	O
them	O
at	O
training	O
time	O
.	O
Even	O
in	O
case	O
of	O
success	O
,	O
the	O
aspect	O
has	O
to	O
exist	O
in	O
the	O
training	O
data	O
.	O
Also	O
,	O
the	O
model	O
is	O
trained	O
with	O
balanced	O
classes	O
,	O
i.e.	O
both	O
supporting	O
and	O
opposing	O
arguments	O
for	O
each	O
topic	O
are	O
seen	O
with	O
equal	O
frequency	O
to	O
prevent	O
possible	O
bias	O
into	O
one	O
or	O
the	O
other	O
direction	O
.	O

To	O
further	O
restrict	O
malicious	O
use	O
,	O
we	O
release	O
the	O
training	O
data	O
for	O
the	O
Arg	O
-	O
CTRLs	O
with	O
an	O
additional	O
clause	O
that	O
forbids	O
use	O
for	O
any	O
other	O
than	O
research	O
purposes	O
.	O
Also	O
,	O
all	O
the	O
training	O
datasets	O
for	O
the	O
Arg	O
-	O
CTRLs	O
will	O
be	O
accessible	O
only	O
via	O
access	O
control	O
(	O
e	O
-	O
mail	O
,	O
name	O
,	O
and	O
purpose	O
of	O
use	O
)	O
.	O
Lastly	O
,	O
this	O
work	O
has	O
been	O
reviewed	O
by	O
the	O
ethics	O
committee	O
of	O
the	O
Technical	O
University	O
of	O
Darmstadt	O
that	O
issued	O
a	O
positive	O
vote	O
.	O
page	O
833	O
-	O
838	O
,	O
USA	O
.	O
American	O
Association	O
for	O
Artificial	O
Intelligence	O
.	O

A	O
Argument	O
Aspect	O
Annotation	O
Study	O

For	O
the	O
final	O
crowdsourcing	O
study	O
,	O
we	O
use	O
Amazon	O
Mechanical	O
Turk	O
.	O
Workers	O
had	O
to	O
take	O
a	O
qualification	O
test	O
,	O
have	O
an	O
acceptance	O
rate	O
of	O
at	O
least	O
95	O
%	O
,	O
and	O
location	O
within	O
the	O
US	O
.	O
We	O
paid	O
$	O
7.6	O
per	O
hour	O
(	O
minimum	O
wage	O
is	O
$	O
7.25	O
per	O
hour	O
)	O
.	O
Each	O
data	O
sample	O
is	O
annotated	O
by	O
eight	O
crowdworkers	O
.	O
In	O
case	O
the	O
ranker	O
cut	O
off	O
the	O
real	O
aspect(s	O
)	O
from	O
the	O
list	O
of	O
candidates	O
,	O
crowdworkers	O
could	O
select	O
any	O
sequence	O
up	O
to	O
four	O
tokens	O
from	O
a	O
second	O
list	O
.	O

Figure	O
2	O
shows	O
the	O
annotation	O
guidelines	O
for	O
the	O
Amazon	O
Mechanical	O
Turk	O
study	O
.	O
Figure	O
3	O
shows	O
one	O
example	O
of	O
a	O
HIT	O
with	O
two	O
aspects	O
selected	O
.	O
Selected	O
aspects	O
are	O
highlighted	O
in	O
the	O
sentence	O
.	O
We	O
did	O
not	O
allow	O
to	O
choose	O
overlapping	O
aspects	O
.	O
If	O
the	O
aspect	O
was	O
not	O
found	O
in	O
the	O
first	O
list	O
provided	O
by	O
the	O
learned	O
ranker	O
,	O
crowdworkers	O
could	O
choose	O
from	O
as	O
second	O
list	O
with	O
the	O
remaining	O
1	O
-	O
4	O
-	O
grams	O
of	O
the	O
sentence	O
(	O
aspect	O
candidates	O
starting	O
or	O
ending	O
with	O
stopwords	O
,	O
as	O
well	O
as	O
candidates	O
with	O
punctuation	O
and	O
numbers	O
,	O
were	O
removed	O
from	O
the	O
list	O
)	O
.	O
Additional	O
checkboxes	O
were	O
added	O
to	O
choose	O
from	O
if	O
the	O
sentence	O
contained	O
no	O
aspect	O
or	O
the	O
aspect	O
was	O
not	O
explicitly	O
mentioned	O
.	O
Figure	O
4	O
shows	O
a	O
ranked	O
list	O
of	O
aspect	O
candidates	O
for	O
an	O
example	O
.	O

The	O
structure	O
of	O
the	O
final	O
dataset	O
is	O
described	O
in	O
Section	O
F.	O
For	O
reproducibility	O
of	O
results	O
,	O
we	O
create	O
fixed	O
splits	O
for	O
in	O
-	O
and	O
cross	O
-	O
topic	O
experiments	O
.	O
9	O
,	O
we	O
show	O
the	O
synonyms	O
used	O
for	O
filtering	O
prior	O
to	O
the	O
argument	O
and	O
stance	O
classification	O
step	O
.	O
We	O
filtered	O
out	O
all	O
sentences	O
that	O
did	O
not	O
contain	O
tokens	O
from	O
the	O
topic	O
they	O
belong	O
to	O
or	O
any	O
synonyms	O
defined	O
for	O
this	O
topic	O
.	O

B	O
Search	O
Query	O
and	O
Topic	O
Relevance	O
Synonyms	O

C	O
Model	O
Parameters	O
and	O
Details	O

All	O
arguments	O
of	O
the	O
training	O
documents	O
are	O
tokenized	O
with	O
a	O
BPE	O
model	O
(	O
Sennrich	O
et	O
al	O
.	O
,	O
2016	O
)	O
trained	O
by	O
the	O
authors	O
of	O
the	O
CTRL	O
(	O
Keskar	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Both	O
the	O
Arg	O
-	O
CTRL	O
CC	O
and	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
are	O
fine	O
-	O
tuned	O
on	O
a	O
Tesla	O
V100	O
with	O
32	O
GB	O
of	O
Memory	O
.	O
We	O
mainly	O
keep	O
the	O
default	O
hyperparameters	O
but	O
reduce	O
the	O
batch	O
size	O
to	O
4	O
and	O
train	O
both	O
models	O
for	O
1	O
epoch	O
.	O
Each	O
model	O
takes	O
around	O
five	O
days	O
to	O
train	O
on	O
the	O
1.6	O
M	O
training	O
sentences	O
.	O

D	O
Reference	O
Data	O
Statistics	O

Table	O
10	O
shows	O
the	O
sources	O
and	O
number	O
of	O
arguments	O
for	O
all	O
topics	O
of	O
the	O
reference	O
dataset	O
.	O
The	O
dataset	O
is	O
used	O
to	O
compare	O
the	O
argument	O
generation	O
models	O
to	O
a	O
retrieval	O
approach	O
.	O

E	O
Examples	O
of	O
Generated	O
Arguments	O

For	O
all	O
eight	O
topics	O
,	O
we	O
show	O
the	O
generated	O
argument	O
with	O
the	O
highest	O
and	O
lowest	O
argument	O
quality	O
score	O
in	O
tables	O
11	O
(	O
Arg	O
-	O
CTRL	O
CC	O
)	O
and	O
12	O
(	O
Arg	O
-	O
CTRL	O
REDDIT	O
)	O
.	O
Text	O
in	O
bold	O
shows	O
the	O
given	O
control	O
code	O
,	O
text	O
afterwards	O
represents	O
the	O
generated	O
argument	O
.	O
Numbers	O
in	O
brackets	O
after	O
the	O
text	O
show	O
the	O
quality	O
score	O
as	O
predicted	O
by	O
the	O
argument	O
quality	O
model	O
.	O

F	O
Argument	O
Aspect	O
Detection	O
Dataset	O

The	O
argument	O
aspect	O
detection	O
dataset	O
contains	O
a	O
total	O
of	O
5,032	O
samples	O
in	O
JSONL	O
-	O
format	O
,	O
i.e.	O
each	O
dataset	O
sample	O
has	O
a	O
separate	O
line	O
and	O
can	O
be	O
parsed	O
as	O
JSON	O
.	O
A	O
sample	O
contains	O
the	O
keys	O
:	O

•	O
hash	O
:	O
Unique	O
identifier	O
.	O

•	O
aspect_pos	O
:	O
List	O
of	O
string	O
tuples	O
"	O
(	O
begin	O
,	O
length	O
)	O
"	O
,	O
marking	O
the	O
character	O
position	O
and	O
length	O
of	O
each	O
aspect	O
within	O
the	O
argument	O
.	O

•	O
aspect_pos_string	O
:	O
The	O
aspects	O
as	O
a	O
list	O
of	O
strings	O
.	O

•	O
stance	O
:	O
Original	O
stance	O
label	O
of	O
the	O
argument	O
towards	O
the	O
topic	O
,	O
taken	O
from	O
the	O
UKP	O
-	O
Corpus	O
(	O
Stab	O
et	O
al	O
.	O
,	O
2018b	O
)	O
.	O
Either	O
"	O
Argument_for	O
"	O
or	O
"	O
Argument_against	O
"	O
.	O

•	O
topic	O
:	O
The	O
topic	O
of	O
the	O
argument	O
.	O

•	O
sentence	O
:	O
The	O
argument	O
.	O

For	O
reproducibility	O
,	O
we	O
define	O
a	O
fixed	O
cross	O
-	O
topic	O
split	O
with	O
the	O
data	O
of	O
two	O
topics	O
as	O
test	O
set	O
(	O
gun	O
control	O
,	O
school	O
uniforms	O
)	O
,	O
the	O
data	O
of	O
one	O
topic	O
as	O
development	O
set	O
(	O
death	O
penalty	O
)	O
,	O
and	O
the	O
data	O
of	O
the	O
remaining	O
five	O
topics	O
as	O
train	O
set	O
.	O
We	O
also	O
create	O
a	O
fixed	O
in	O
-	O
topic	O
split	O
by	O
randomly	O
taking	O
3,532	O
samples	O
of	O
all	O
topics	O
for	O
training	O
,	O
500	O
for	O
development	O
,	O
and	O
1,000	O
for	O
testing	O
.	O
nuclear	O
AND	O
(	O
energy	O
OR	O
fission	O
OR	O
power	O
OR	O
plant	O
)	O
In	O
addition	O
,	O
we	O
must	O
continue	O
developing	O
safer	O
technologies	O
like	O
small	O
modular	O
reactors	O
which	O
will	O
help	O
us	O
meet	O
our	O
nation	O
's	O
need	O
for	O
reliable	O
,	O
emission	O
-	O
free	O
sources	O
of	O
low	O
-	O
emission	O
energy	O
while	O
also	O
creating	O
jobs	O
and	O
providing	O
solutions	O
to	O
some	O
of	O
the	O
world	O
s	O
most	O
pressing	O
problems	O
:	O
climate	O
change	O
,	O
food	O
security	O
and	O
sustainable	O
development	O
.	O
(	O
0.96	O
)	O
nuclear	O
energy	O
CON	O
leak	O
.	O
"	O
We	O
are	O
concerned	O
about	O
the	O
possibility	O
of	O
further	O
releases	O
of	O
radioactivity	O
due	O
to	O
possible	O
melting	O
or	O
cracking	O
of	O
fuel	O
rods	O
at	O
the	O
No	O
.	O
(	O
0.47	O
)	O
death	O
penalty	O
CON	O
inhuman	O
.	O
Amnesty	O
International	O
opposes	O
the	O
death	O
penalty	O
in	O
all	O
cases	O
as	O
the	O
ultimate	O
form	O
of	O
cruel	O
,	O
inhuman	O
or	O
degrading	O
punishment	O
and	O
a	O
violation	O
of	O
fundamental	O
rights	O
-the	O
right	O
to	O
life	O
and	O
the	O
prohibition	O
of	O
torture	O
.	O

(	O
1.00	O
)	O
death	O
penalty	O
CON	O
god	O
.	O
And	O
yet	O
,	O
while	O
the	O
Church	O
exhorts	O
civil	O
authorities	O
.	O
(	O
0.41	O
)	O
school	O
uniforms	O
CON	O
resources	O
.	O
The	O
lack	O
of	O
a	O
uniform	O
system	O
for	O
the	O
collection	O
and	O
use	O
of	O
data	O
,	O
as	O
well	O
as	O
insufficient	O
funding	O
to	O
collect	O
it	O
,	O
are	O
major	O
barriers	O
that	O
limit	O
access	O
to	O
information	O
on	O
student	O
achievement	O
in	O
schools	O
with	O
high	O
concentrations	O
of	O
students	O
from	O
low	O
-	O
income	O
families	O
.	O
(	O
0.92	O
)	O
school	O
uniforms	O
PRO	O
fun	O
:	O
The	O
kids	O
are	O
having	O
a	O
lot	O
of	O
fun	O
.	O
(	O
0.38	O
)	O
gun	O
control	O
PRO	O
homicides	O
.	O
In	O
addition	O
to	O
being	O
an	O
effective	O
crime	O
deterrent	O
and	O
reducing	O
suicides	O
,	O
research	O
has	O
shown	O
that	O
defensive	O
firearm	O
use	O
prevents	O
more	O
than	O
80	O
percent	O
of	O
all	O
gun	O
-	O
related	O
violent	O
crimes	O
including	O
homicides	O
,	O
robberies	O
,	O
assaults	O
,	O
rapes	O
,	O
and	O
aggravated	O
assaults	O
[	O
1	O
]	O
.	O
(	O
0.96	O
)	O
gun	O
control	O
PRO	O
prevent	O
;	O
You	O
liberals	O
need	O
to	O
wake	O
up	O
and	O
smell	O
the	O
coffee	O
.	O
(	O
0.43	O
)	O
marijuana	O
legalization	O
PRO	O
safer	O
:	O
Legalizing	O
cannabis	O
will	O
help	O
reduce	O
crime	O
rates	O
(	O
especially	O
violent	O
crimes	O
)	O
and	O
make	O
society	O
safer	O
overall	O
.	O
(	O
0.96	O
)	O
marijuana	O
legalization	O
PRO	O
benefits	O
.	O
Decrease	O
amount	O
of	O
police	O
officers	O
needed	O
6	O
.	O
(	O
0.37	O
)	O
minimum	O
wage	O
PRO	O
poor	O
:	O
Raising	O
the	O
minimum	O
wage	O
will	O
not	O
only	O
benefit	O
those	O
working	O
full	O
time	O
but	O
also	O
reduce	O
government	O
expenditures	O
on	O
social	O
services	O
such	O
as	O
food	O
stamps	O
and	O
Medicaid	O
which	O
disproportionately	O
affect	O
the	O
poor	O
.	O
(	O
0.97	O
)	O
minimum	O
wage	O
CON	O
cost	O
:	O
If	O
you	O
raise	O
the	O
price	O
of	O
a	O
Big	O
Mac	O
to	O
$	O
10	O
and	O
then	O
pay	O
an	O
extra	O
dime	O
or	O
two	O
per	O
burger	O
so	O
that	O
it	O
's	O
still	O
only	O
$	O
9	O
...	O
well	O
,	O
maybe	O
your	O
business	O
is	O
n't	O
worth	O
saving	O
at	O
all	O
because	O
there	O
just	O
wo	O
n't	O
be	O
anyone	O
willing	O
to	O
eat	O
out	O
after	O
they	O
'	O
ve	O
had	O
their	O
fill	O
(	O
unless	O
perhaps	O
they	O
're	O
on	O
welfare	O
)	O
and	O
now	O
have	O
no	O
job	O
and	O
ca	O
n't	O
afford	O
health	O
care	O
eitherbut	O
I	O
doubt	O
very	O
seriously	O
that	O
most	O
people	O
making	O
those	O
kinds	O
of	O
wages	O
/	O
salaries	O
would	O
suddenly	O
decide	O
not	O
to	O
work	O
if	O
forced	O
to	O
shoulder	O
the	O
added	O
cost	O
.	O
(	O
0.44	O
)	O
12	O
:	O
Generated	O
arguments	O
with	O
the	O
Arg	O
-	O
CTRL	O
REDDIT	O
.	O
Text	O
in	O
bold	O
shows	O
the	O
given	O
control	O
code	O
,	O
text	O
afterwards	O
represents	O
the	O
generated	O
argument	O
.	O
Numbers	O
in	O
brackets	O
after	O
the	O
text	O
show	O
the	O
quality	O
score	O
as	O
predicted	O
by	O
the	O
argument	O
quality	O
model	O
.	O

Acknowledgements	O

We	O
thank	O
Tilman	O
Beck	O
and	O
Nandan	O
Thakur	O
for	O
their	O
support	O
in	O
the	O
human	O
evaluation	O
(	O
Section	O
7.1	O
)	O
.	O
This	O
work	O
has	O
been	O
supported	O
by	O
the	O
German	O
Research	O
Foundation	O
within	O
the	O
project	O
"	O
Open	O
Argument	O
Mining	O
"	O
(	O
GU	O
798/25	O
-	O
1	O
)	O
,	O
associated	O
with	O
the	O
Priority	O
Program	O
"	O
Robust	O
Argumentation	O
Machines	O
(	O
RATIO	O
)	O
"	O
(	O
SPP-1999	O
)	O
.	O

Linking	O
Entities	O
to	O
Unseen	O
Knowledge	O
Bases	O
with	O
Arbitrary	O
Schemas	O

In	O
entity	O
linking	O
,	O
mentions	O
of	O
named	O
entities	O
in	O
raw	O
text	O
are	O
disambiguated	O
against	O
a	O
knowledge	O
base	O
(	O
KB	O
)	O
.	O
This	O
work	O
focuses	O
on	O
linking	O
to	O
unseen	O
KBs	O
that	O
do	O
not	O
have	O
training	O
data	O
and	O
whose	O
schema	O
is	O
unknown	O
during	O
training	O
.	O
Our	O
approach	O
relies	O
on	O
methods	O
to	O
flexibly	O
convert	O
entities	O
with	O
several	O
attribute	O
-	O
value	O
pairs	O
from	O
arbitrary	O
KBs	O
into	O
flat	O
strings	O
,	O
which	O
we	O
use	O
in	O
conjunction	O
with	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
models	O
for	O
zero	O
-	O
shot	O
linking	O
.	O
We	O
further	O
improve	O
the	O
generalization	O
of	O
our	O
model	O
using	O
two	O
regularization	O
schemes	O
based	O
on	O
shuffling	O
of	O
entity	O
attributes	O
and	O
handling	O
of	O
unseen	O
attributes	O
.	O
Experiments	O
on	O
English	O
datasets	O
where	O
models	O
are	O
trained	O
on	O
the	O
CoNLL	O
dataset	O
,	O
and	O
tested	O
on	O
the	O
TAC	O
-	O
KBP	O
2010	O
dataset	O
show	O
that	O
our	O
models	O
are	O
12	O
%	O
(	O
absolute	O
)	O
more	O
accurate	O
than	O
baseline	O
models	O
that	O
simply	O
flatten	O
entities	O
from	O
the	O
target	O
KB	O
.	O
Unlike	O
prior	O
work	O
,	O
our	O
approach	O
also	O
allows	O
for	O
seamlessly	O
combining	O
multiple	O
training	O
datasets	O
.	O
We	O
test	O
this	O
ability	O
by	O
adding	O
both	O
a	O
completely	O
different	O
dataset	O
(	O
Wikia	O
)	O
,	O
as	O
well	O
as	O
increasing	O
amount	O
of	O
training	O
data	O
from	O
the	O
TAC	O
-	O
KBP	O
2010	O
training	O
set	O
.	O
Our	O
models	O
are	O
more	O
accurate	O
across	O
the	O
board	O
compared	O
to	O
baselines	O
.	O

Introduction	O

Entity	O
linking	O
consists	O
of	O
linking	O
mentions	O
of	O
entities	O
found	O
in	O
text	O
against	O
canonical	O
entities	O
found	O
in	O
a	O
target	O
knowledge	O
base	O
(	O
KB	O
)	O
.	O
Early	O
work	O
in	O
this	O
area	O
was	O
motivated	O
by	O
the	O
availability	O
of	O
large	O
KBs	O
with	O
millions	O
of	O
entities	O
(	O
Bunescu	O
and	O
Paşca	O
,	O
2006	O
)	O
.	O
Most	O
subsequent	O
work	O
has	O
followed	O
this	O
tradition	O
of	O
linking	O
to	O
a	O
handful	O
of	O
large	O
,	O
publicly	O
available	O
KBs	O
such	O
as	O
Wikipedia	O
,	O
DBPedia	O
(	O
Auer	O
et	O
al	O
.	O
,	O
2007	O
)	O
or	O
the	O
KBs	O
used	O
in	O
the	O
now	O
decade	O
-	O
old	O
TAC	O
-	O
KBP	O
challenges	O
(	O
McNamee	O
and	O
Dang	O
,	O
2009;Ji	O
et	O
al	O
.	O
,	O
2010	O
)	O
.	O
As	O
a	O
result	O
,	O
previous	O
work	O
always	O
assumes	O
complete	O
knowledge	O
of	O
the	O
schema	O
of	O
the	O
target	O
KB	O
that	O
entity	O
linking	O
models	O
are	O
trained	O
for	O
,	O
i.e.	O
how	O
many	O
and	O
which	O
attributes	O
are	O
used	O
to	O
represent	O
entities	O
in	O
the	O
KB	O
.	O
This	O
allows	O
training	O
supervised	O
machine	O
learning	O
models	O
that	O
exploit	O
the	O
schema	O
along	O
with	O
labeled	O
data	O
that	O
link	O
mentions	O
to	O
this	O
a	O
priori	O
known	O
KB	O
.	O
However	O
,	O
this	O
strong	O
assumption	O
breaks	O
down	O
in	O
scenarios	O
which	O
require	O
linking	O
to	O
KBs	O
that	O
are	O
not	O
known	O
at	O
training	O
time	O
.	O
For	O
example	O
,	O
a	O
company	O
might	O
want	O
to	O
automatically	O
link	O
mentions	O
of	O
its	O
products	O
to	O
an	O
internal	O
KB	O
of	O
products	O
that	O
has	O
a	O
rich	O
schema	O
with	O
several	O
attributes	O
such	O
as	O
product	O
category	O
,	O
description	O
,	O
dimensions	O
,	O
etc	O
.	O
It	O
is	O
very	O
unlikely	O
that	O
the	O
company	O
will	O
have	O
training	O
data	O
of	O
this	O
nature	O
,	O
i.e.	O
mentions	O
of	O
products	O
linked	O
to	O
its	O
database	O
.	O

Our	O
focus	O
is	O
on	O
linking	O
entities	O
to	O
unseen	O
KBs	O
with	O
arbitrary	O
schemas	O
.	O
One	O
solution	O
is	O
to	O
annotate	O
data	O
that	O
can	O
be	O
used	O
to	O
train	O
specialized	O
models	O
for	O
each	O
target	O
KB	O
of	O
interest	O
,	O
but	O
this	O
is	O
not	O
scalable	O
.	O
A	O
more	O
generic	O
solution	O
is	O
to	O
build	O
entity	O
linking	O
models	O
that	O
work	O
with	O
arbitrary	O
KBs	O
.	O
We	O
follow	O
this	O
latter	O
approach	O
and	O
build	O
entity	O
linking	O
models	O
that	O
link	O
to	O
target	O
KBs	O
that	O
have	O
not	O
been	O
observed	O
during	O
training	O
.	O
1	O
Our	O
solution	O
builds	O
on	O
recent	O
models	O
for	O
zero	O
-	O
shot	O
entity	O
linking	O
(	O
Wu	O
et	O
al	O
.	O
,	O
2020;Logeswaran	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
However	O
,	O
these	O
models	O
assume	O
the	O
same	O
,	O
simple	O
KB	O
schema	O
during	O
training	O
and	O
inference	O
.	O
We	O
generalize	O
these	O
models	O
to	O
handle	O
different	O
KBs	O
during	O
training	O
and	O
inference	O
,	O
containing	O
entities	O
represented	O
with	O
an	O
arbitrary	O
set	O
of	O
attribute	O
-	O
value	O
pairs	O
.	O
This	O
generalization	O
relies	O
on	O
two	O
key	O
ideas	O
.	O
First	O
,	O
we	O
convert	O
KB	O
entities	O
into	O
strings	O
that	O
are	O
consumed	O
by	O
the	O
models	O
for	O
zero	O
-	O
shot	O
linking	O
.	O
Central	O
to	O
the	O
string	O
representation	O
are	O
special	O
tokens	O
called	O
attribute	O
separators	O
,	O
which	O
represent	O
frequently	O
occurring	O
attributes	O
in	O
the	O
training	O
KB(s	O
)	O
,	O
and	O
carry	O
over	O
their	O
knowledge	O
to	O
unseen	O
KBs	O
during	O
inference	O
(	O
Section	O
4.1	O
)	O
.	O
Second	O
,	O
we	O
generate	O
more	O
flexible	O
string	O
representations	O
by	O
shuffling	O
entity	O
attributes	O
before	O
converting	O
them	O
to	O
strings	O
,	O

Generic	O
EL	O

Zero	O
-	O
shot	O
EL	O
Linking	O
to	O
any	O
DB	O
This	O
work	O
(	O
Logeswaran	O
et	O
al	O
.	O
,	O
2019	O
)	O
(	O
Sil	O
et	O
al	O
.	O
,	O
2012	O
)	O
Test	O
entities	O
not	O
seen	O
during	O
training	O
Test	O
KB	O
schema	O
unknown	O
Out	O
-	O
of	O
-	O
domain	O
test	O
data	O
Unrestricted	O
Candidate	O
Set	O
and	O
by	O
stochastically	O
removing	O
attribute	O
separators	O
to	O
generalize	O
to	O
unseen	O
attributes	O
(	O
Section	O
4.2	O
)	O
.	O

Our	O
primary	O
experiments	O
are	O
cross	O
-	O
KB	O
and	O
focus	O
on	O
English	O
datasets	O
.	O
We	O
train	O
models	O
to	O
link	O
to	O
one	O
KB	O
during	O
training	O
(	O
viz	O
.	O
Wikidata	O
)	O
,	O
and	O
evaluate	O
them	O
for	O
their	O
ability	O
to	O
link	O
to	O
an	O
unseen	O
KB	O
(	O
viz	O
.	O
the	O
TAC	O
-	O
KBP	O
Knowledge	O
Base	O
)	O
.	O
These	O
experiments	O
reveal	O
that	O
our	O
model	O
with	O
attributeseparators	O
and	O
the	O
two	O
generalization	O
schemes	O
are	O
12	O
-	O
14	O
%	O
more	O
accurate	O
than	O
the	O
baseline	O
zero	O
-	O
shot	O
models	O
.	O
Ablation	O
studies	O
reveal	O
that	O
all	O
components	O
individually	O
contribute	O
to	O
this	O
improvement	O
,	O
but	O
combining	O
all	O
of	O
them	O
yields	O
the	O
most	O
accurate	O
models	O
.	O

Unlike	O
previous	O
work	O
,	O
our	O
models	O
also	O
allow	O
seamless	O
mixing	O
of	O
multiple	O
training	O
datasets	O
which	O
link	O
to	O
different	O
KBs	O
with	O
different	O
schemas	O
.	O
We	O
investigate	O
the	O
impact	O
of	O
training	O
on	O
multiple	O
datasets	O
in	O
two	O
sets	O
of	O
experiments	O
involving	O
additional	O
training	O
data	O
that	O
links	O
to	O
(	O
a	O
)	O
a	O
third	O
KB	O
that	O
is	O
different	O
from	O
our	O
original	O
training	O
and	O
testing	O
KBs	O
,	O
and	O
(	O
b	O
)	O
the	O
same	O
KB	O
as	O
the	O
test	O
data	O
.	O
These	O
experiments	O
reveal	O
that	O
our	O
models	O
perform	O
favorably	O
under	O
all	O
conditions	O
compared	O
to	O
baselines	O
.	O

Background	O

Conventional	O
entity	O
linking	O
models	O
are	O
trained	O
and	O
evaluated	O
on	O
the	O
same	O
KB	O
,	O
which	O
is	O
typically	O
Wikipedia	O
,	O
or	O
derived	O
from	O
Wikipedia	O
(	O
Bunescu	O
and	O
Paşca	O
,	O
2006	O
;	O
.	O
This	O
limited	O
scope	O
allows	O
models	O
to	O
use	O
other	O
sources	O
of	O
information	O
to	O
improve	O
linking	O
,	O
including	O
alias	O
tables	O
,	O
frequency	O
statistics	O
,	O
and	O
rich	O
metadata	O
.	O

Beyond	O
Conventional	O
Entity	O
Linking	O
There	O
have	O
been	O
several	O
attempts	O
to	O
go	O
beyond	O
such	O
conventional	O
settings	O
,	O
e.g.	O
by	O
linking	O
to	O
KBs	O
from	O
diverse	O
domains	O
such	O
as	O
the	O
biomedical	O
sciences	O
(	O
Zheng	O
et	O
al	O
.	O
,	O
2014;D'Souza	O
and	O
Ng	O
,	O
2015	O
)	O
and	O
music	O
(	O
Oramas	O
et	O
al	O
.	O
,	O
2016	O
)	O
or	O
even	O
being	O
completely	O
domain	O
and	O
language	O
independent	O
Onoe	O
and	O
Durrett	O
,	O
2020	O
)	O
.	O
Lin	O
et	O
al	O
.	O
(	O
2017	O
)	O
discuss	O
approaches	O
to	O
link	O
entities	O
to	O
a	O
KB	O
that	O
simply	O
contains	O
a	O
list	O
of	O
names	O
without	O
any	O
other	O
information	O
.	O
Sil	O
et	O
al	O
.	O
(	O
2012	O
)	O
use	O
databaseagnostic	O
features	O
to	O
link	O
against	O
arbitrary	O
databases	O
.	O
However	O
,	O
their	O
approach	O
still	O
requires	O
training	O
data	O
from	O
the	O
target	O
KB	O
.	O
In	O
contrast	O
,	O
this	O
work	O
aims	O
to	O
train	O
entity	O
linking	O
models	O
that	O
do	O
not	O
rely	O
on	O
training	O
data	O
from	O
the	O
target	O
KB	O
,	O
and	O
can	O
be	O
trained	O
on	O
arbitrary	O
KBs	O
,	O
and	O
applied	O
to	O
a	O
different	O
set	O
of	O
KBs	O
.	O
Pan	O
et	O
al	O
.	O
(	O
2015	O
)	O
also	O
do	O
unsupervised	O
entity	O
linking	O
by	O
generating	O
rich	O
context	O
representations	O
for	O
mentions	O
using	O
Abstract	O
Meaning	O
Representations	O
(	O
Banarescu	O
et	O
al	O
.	O
,	O
2013	O
)	O
,	O
followed	O
by	O
unsupervised	O
graph	O
inference	O
to	O
compare	O
contexts	O
.	O
They	O
assume	O
a	O
rich	O
target	O
KB	O
that	O
can	O
be	O
converted	O
to	O
a	O
connected	O
graph	O
.	O
This	O
works	O
for	O
Wikipedia	O
and	O
adjacent	O
resources	O
but	O
not	O
for	O
arbitrary	O
KBs	O
.	O
Logeswaran	O
et	O
al	O
.	O
(	O
2019	O
)	O
introduce	O
a	O
novel	O
zeroshot	O
framework	O
to	O
"	O
develop	O
entity	O
linking	O
systems	O
that	O
can	O
generalize	O
to	O
unseen	O
specialized	O
entities	O
"	O
.	O
Table	O
1	O
summarizes	O
differences	O
between	O
our	O
framework	O
and	O
those	O
from	O
prior	O
work	O
.	O

Contextualized	O
Representations	O
for	O
Entity	O
Linking	O
Models	O
in	O
this	O
work	O
are	O
based	O
on	O
BERT	O
.	O
While	O
many	O
studies	O
have	O
tried	O
to	O
explain	O
the	O
effectiveness	O
of	O
BERT	O
for	O
NLP	O
tasks	O
(	O
Rogers	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
the	O
work	O
by	O
Tenney	O
et	O
al	O
.	O
(	O
2019	O
)	O
is	O
most	O
relevant	O
as	O
they	O
use	O
probing	O
tasks	O
to	O
show	O
that	O
BERT	O
encodes	O
knowledge	O
of	O
entities	O
.	O
This	O
has	O
also	O
been	O
shown	O
empirically	O
by	O
many	O
works	O
that	O
use	O
BERT	O
and	O
other	O
contextualized	O
models	O
for	O
entity	O
linking	O
and	O
disambiguation	O
(	O
Broscheit	O
,	O
2019;Shahbazi	O
et	O
al	O
.	O
,	O
2019;Yamada	O
et	O
al	O
.	O
,	O
2020;Févry	O
et	O
al	O
.	O
,	O
2020;Poerner	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

Preliminaries	O

Entity	O
Linking	O
Setup	O

Entity	O
linking	O
consists	O
of	O
disambiguating	O
entity	O
mentions	O
M	O
from	O
one	O
or	O
more	O
documents	O
to	O
a	O
target	O
knowledge	O
base	O
,	O
KB	O
,	O
containing	O
unique	O
entities	O
.	O
We	O
assume	O
that	O
each	O
entity	O
e	O
∈	O
KB	O
is	O
represented	O
using	O
a	O
set	O
of	O
attribute	O
-	O
value	O
pairs	O

{	O
(	O
k	O
i	O
,	O
v	O
i	O
)	O
}	O
n	O
i=1	O
.	O

The	O
attributes	O
k	O
i	O
collectively	O
form	O
the	O
schema	O
of	O
KB	O
.	O
The	O
disambiguation	O
of	O
each	O
m	O
∈	O
M	O
is	O
aided	O
by	O
the	O
context	O
c	O
in	O
which	O
m	O
appears	O
.	O

Models	O
for	O
entity	O
linking	O
typically	O
consist	O
of	O
two	O
stages	O
that	O
balance	O
recall	O
and	O
precision	O
.	O

Typically	O
,	O
models	O
for	O
candidate	O
generation	O
are	O
less	O
complex	O
(	O
and	O
hence	O
,	O
less	O
precise	O
)	O
than	O
those	O
used	O
in	O
the	O
following	O
(	O
re	O
-	O
ranking	O
)	O
stage	O
since	O
they	O
handle	O
all	O
entities	O
in	O
KB	O
.	O

1	O
.	O
Candidate	O
generation	O
:	O
The	O
objective	O
of	O
this	O
stage	O
is	O
to	O
select	O
K	O
candidate	O
entities	O
E	O
⊂	O
KB	O
for	O
each	O
mention	O
m	O
∈	O
M	O
,	O
where	O
K	O
is	O
a	O
hyperparameter	O
and	O
K	O
<	O
<	O
|KB|	O
.	O

Instead	O
,	O
the	O
goal	O
of	O
these	O
models	O
is	O
to	O
produce	O
a	O
small	O
but	O
high	O
-	O
recall	O
candidate	O
list	O
E.	O
Ergo	O
,	O
the	O
success	O
of	O
this	O
stage	O
is	O
measured	O
using	O
a	O
metric	O
such	O
as	O
recall@K	O
i.e.	O
whether	O
the	O
candidate	O
list	O
contains	O
the	O
correct	O
entity	O
.	O

2	O
.	O
Candidate	O
Reranking	O
:	O
This	O
stage	O
ranks	O
the	O
candidates	O
in	O
E	O
by	O
how	O
likely	O
they	O
are	O
to	O
be	O
the	O
correct	O
entity	O
.	O
Unlike	O
candidate	O
generation	O
,	O
models	O
for	O
re	O
-	O
ranking	O
are	O
typically	O
more	O
complex	O
and	O
oriented	O
towards	O
generating	O
a	O
high	O
-	O
precision	O
ranked	O
list	O
since	O
the	O
objective	O
of	O
this	O
stage	O
is	O
to	O
identify	O
the	O
most	O
likely	O
entity	O
for	O
each	O
mention	O
.	O
This	O
stage	O
is	O
evaluated	O
using	O
precision@1	O
(	O
or	O
accuracy	O
)	O
i.e.	O
whether	O
the	O
highest	O
ranked	O
entity	O
is	O
the	O
correct	O
entity	O
.	O

In	O
traditional	O
entity	O
linking	O
,	O
the	O
training	O
mentions	O
M	O
train	O
and	O
test	O
mentions	O
M	O
test	O
both	O
link	O
to	O
the	O
same	O
KB	O
.	O
Even	O
in	O
the	O
zero	O
-	O
shot	O
settings	O
of	O
Logeswaran	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
while	O
the	O
training	O
and	O
target	O
domains	O
and	O
KBs	O
are	O
mutually	O
exclusive	O
,	O
the	O
schema	O
of	O
the	O
KB	O
is	O
constant	O
and	O
known	O
.	O
On	O
the	O
contrary	O
,	O
our	O
goal	O
is	O
to	O
link	O
test	O
mentions	O
M	O
test	O
to	O
a	O
knowledge	O
base	O
KB	O
test	O
which	O
is	O
not	O
known	O
during	O
training	O
.	O
The	O
objective	O
is	O
to	O
train	O
models	O
on	O
mentions	O
M	O
train	O
that	O
link	O
to	O
KB	O
train	O
and	O
directly	O
use	O
these	O
models	O
to	O
link	O
M	O
test	O
to	O
KB	O
test	O
.	O

Zero	O
-	O
shot	O
Entity	O
Linking	O

The	O
starting	O
point	O
(	O
and	O
baselines	O
)	O
for	O
our	O
work	O
are	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
models	O
for	O
zero	O
-	O
shot	O
entity	O
linking	O
,	O
which	O
we	O
briefly	O
describe	O
here	O
(	O
Wu	O
et	O
al	O
.	O
,	O
2020;Logeswaran	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
2	O
Candidate	O
Generation	O
Our	O
baseline	O
candidate	O
generation	O
approach	O
relies	O
on	O
similarities	O
between	O
mentions	O
and	O
candidates	O
in	O
a	O
vector	O
space	O
to	O
identify	O
the	O
candidates	O
for	O
each	O
mention	O
(	O
Wu	O
et	O
al	O
.	O
,	O
2020	O
)	O
using	O
two	O
BERT	O
models	O
.	O
The	O
first	O
BERT	O
model	O
encodes	O
a	O
mention	O
m	O
along	O
with	O
its	O
context	O
c	O
into	O
a	O
vector	O
representation	O
v	O
m	O
.	O
v	O
m	O
is	O
obtained	O
from	O
the	O
pooled	O
representation	O
captured	O
by	O
the	O
[	O
CLS	O
]	O
token	O
used	O
in	O
BERT	O
models	O
to	O
indicate	O
the	O
start	O
of	O
a	O
sequence	O
.	O
In	O
this	O
encoder	O
,	O
a	O
binary	O
(	O
0/1	O
)	O
indicator	O
vector	O
is	O
used	O
to	O
identify	O
the	O
mention	O
span	O
.	O
The	O
embeddings	O
for	O
this	O
indicator	O
vector	O
(	O
indicator	O
embeddings	O
)	O
are	O
added	O
to	O
the	O
token	O
embeddings	O
of	O
the	O
mention	O
as	O
in	O
Logeswaran	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O

The	O
second	O
unmodified	O
BERT	O
model	O
(	O
i.e.	O
not	O
containing	O
the	O
indicator	O
embeddings	O
as	O
in	O
the	O
mention	O
encoder	O
)	O
independently	O
encodes	O
each	O
e	O
∈	O
KB	O
into	O
vectors	O
.	O
The	O
candidates	O
E	O
for	O
a	O
mention	O
are	O
the	O
K	O
entities	O
whose	O
representations	O
are	O
most	O
similar	O
to	O
v	O
m	O
.	O
Both	O
BERT	O
models	O
are	O
fine	O
-	O
tuned	O
jointly	O
using	O
a	O
cross	O
-	O
entropy	O
loss	O
to	O
maximize	O
the	O
similarity	O
between	O
a	O
mention	O
and	O
its	O
corresponding	O
correct	O
entity	O
,	O
when	O
compared	O
to	O
other	O
random	O
entities	O
.	O

Candidate	O
Re	O
-	O
ranking	O
The	O
candidate	O
reranking	O
approach	O
uses	O
a	O
BERT	O
-	O
based	O
crossattention	O
encoder	O
to	O
jointly	O
encode	O
a	O
mention	O
and	O
its	O
context	O
along	O
with	O
each	O
candidate	O
from	O
E	O
(	O
Logeswaran	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Specifically	O
,	O
the	O
mention	O
m	O
is	O
concatenated	O
with	O
its	O
context	O
on	O
the	O
left	O
(	O
c	O
l	O
)	O
,	O
its	O
context	O
on	O
the	O
right	O
(	O
c	O
r	O
)	O
,	O
and	O
a	O
single	O
candidate	O
entity	O
e	O
∈	O
E.	O
An	O
[	O
SEP	O
]	O
token	O
,	O
which	O
is	O
used	O
in	O
BERT	O
to	O
separate	O
inputs	O
from	O
different	O
segments	O
,	O
is	O
used	O
here	O
to	O
separate	O
the	O
mention	O
in	O
context	O
,	O
from	O
the	O
candidate	O
.	O
This	O
concatenated	O
string	O
is	O
encoded	O
using	O
BERT	O
3	O
to	O
obtain	O
,	O
h	O
m	O
,	O
e	O
a	O
representation	O
for	O
this	O
mention	O
/	O
candidate	O
pair	O
(	O
from	O
the	O
[	O
CLS	O
]	O
token	O
)	O
.	O
Given	O
a	O
candidate	O
list	O
E	O
of	O
size	O
K	O
generated	O
in	O
the	O
previous	O
stage	O
,	O
K	O
scores	O
are	O
generated	O
for	O
each	O
mention	O
,	O
which	O
are	O
subsequently	O
scored	O
using	O
a	O
dot	O
-	O
product	O
with	O
a	O
learned	O
weight	O
vector	O
(	O
w	O
)	O
.	O
Thus	O
,	O

h	O
m	O
,	O
e	O
=	O
BERT([CLS	O
]	O
c	O
l	O
m	O
c	O
r	O
[	O
SEP	O
]	O
e	O
[	O
SEP	O
]	O
)	O
,	O
score	O
m	O
,	O
e	O
=	O
w	O
T	O
h	O
m	O
,	O
e	O
.	O

The	O
candidate	O
with	O
the	O
highest	O
score	O
is	O
chosen	O
as	O
the	O
correct	O
entity	O
,	O
i.e.	O

Linking	O
to	O
Unseen	O
Knowledge	O
Bases	O

The	O
models	O
in	O
Section	O
3	O
were	O
designed	O
to	O
operate	O
in	O
settings	O
where	O
the	O
entities	O
in	O
the	O
target	O
KB	O
were	O
only	O
represented	O
using	O
a	O
textual	O
description	O
.	O
For	O
example	O
,	O
the	O
entity	O
Douglas	O
Adams	O
would	O
be	O
represented	O
in	O
such	O
a	O
database	O
using	O
a	O
description	O
as	O
follows	O
:	O
"	O
Douglas	O
Adams	O
was	O
an	O
English	O
author	O
,	O
screenwriter	O
,	O
essayist	O
,	O
humorist	O
,	O
satirist	O
and	O
dramatist	O
.	O
He	O
was	O
the	O
author	O
of	O
The	O
Hitchhiker	O
's	O
Guide	O
to	O
the	O
Galaxy	O
.	O
"	O

However	O
,	O
linking	O
to	O
unseen	O
KBs	O
requires	O
handling	O
entities	O
with	O
an	O
arbitrary	O
number	O
and	O
type	O
of	O
attributes	O
.	O
The	O
same	O
entity	O
(	O
Douglas	O
Adams	O
)	O
can	O
be	O
represented	O
in	O
a	O
different	O
KB	O
using	O
attributes	O
such	O
as	O
"	O
name	O
"	O
,	O
"	O
place	O
of	O
birth	O
"	O
,	O
etc	O
.	O
(	O
top	O
of	O
Figure	O
1	O
)	O
.	O
This	O
raises	O
the	O
question	O
of	O
whether	O
such	O
models	O
,	O
that	O
harness	O
the	O
power	O
of	O
pre	O
-	O
trained	O
language	O
models	O
,	O
generalize	O
to	O
linking	O
mentions	O
to	O
unseen	O
KBs	O
,	O
including	O
those	O
without	O
such	O
textual	O
descriptions	O
.	O
This	O
section	O
presents	O
multiple	O
ideas	O
to	O
this	O
end	O
.	O

Representing	O
Arbitrary	O
Entities	O
using	O

Attribute	O
Separators	O

One	O
way	O
of	O
using	O
these	O
models	O
for	O
linking	O
against	O
arbitrary	O
KBs	O
is	O
by	O
defining	O
an	O
attribute	O
-	O
to	O
-	O
text	O
function	O
f	O
,	O
that	O
maps	O
arbitrary	O
entities	O
with	O
any	O
set	O
of	O
attributes	O
{	O
k	O
i	O
,	O
v	O
i	O
}	O
n	O
i=1	O
to	O
a	O
string	O
representation	O
e	O
that	O
can	O
be	O
consumed	O
by	O
BERT	O
,	O
i.e.	O

e	O
=	O
f	O
(	O
{	O
k	O
i	O
,	O
v	O
i	O
}	O
n	O
i=1	O
)	O
.	O

If	O
all	O
entities	O
in	O
the	O
KB	O
are	O
represented	O
using	O
such	O
string	O
representations	O
,	O
then	O
the	O
models	O
described	O
in	O
Section	O
3	O
can	O
directly	O
be	O
used	O
for	O
arbitrary	O
schemas	O
.	O
This	O
leads	O
to	O
the	O
question	O
:	O
how	O
can	O
we	O
generate	O
string	O
representations	O
for	O
entities	O
from	O
arbitrary	O
KBs	O
such	O
that	O
they	O
can	O
be	O
used	O
for	O
BERT	O
-	O
based	O
models	O
?	O
Alternatively	O
,	O
what	O
form	O
can	O
f	O
take	O
?	O

A	O
simple	O
answer	O
to	O
this	O
question	O
is	O
concatenation	O
of	O
the	O
values	O
v	O
i	O
,	O
given	O
by	O

f	O
(	O
{	O
k	O
i	O
,	O
v	O
i	O
}	O
n	O
i=1	O
)	O
=	O
v	O
1	O
v	O
2	O
...	O
v	O
n	O
.	O

We	O
can	O
improve	O
on	O
this	O
by	O
adding	O
some	O
structure	O
to	O
this	O
representation	O
by	O
teaching	O
our	O
model	O
that	O
the	O
v	O
i	O
belong	O
to	O
different	O
segments	O
.	O
As	O
in	O
the	O
baseline	O
candidate	O
re	O
-	O
ranking	O
model	O
,	O
we	O
do	O
this	O
by	O
separating	O
them	O
with	O
[	O
SEP	O
]	O
tokens	O
.	O
We	O
call	O
this	O
[	O
SEP]-separation	O
.	O
This	O
approach	O
is	O
also	O
used	O
by	O
Logeswaran	O
et	O
al	O
.	O
(	O
2019	O
)	O
andMulang	O
'	O
et	O
al	O
.	O
(	O
2020	O
)	O
"	O
name	O
"	O
:	O
"	O
Douglas	O
Adams	O
"	O
"	O
place	O
of	O
birth	O
"	O
:	O
"	O
Cambridge	O
"	O
"	O
occupation	O
"	O
:	O
"	O
novelist	O
"	O
"	O
employer	O
"	O
:	O
"	O
BBC	O
"	O
to	O
separate	O
the	O
entity	O
attributes	O
in	O
their	O
respective	O
KBs	O
.	O

f	O
(	O
{	O
k	O
i	O
,	O
v	O
i	O
}	O
n	O
i=1	O
)	O
=	O
[	O
SEP	O
]	O
v	O
1	O
[	O
SEP	O
]	O
v	O
2	O
...	O
[	O
SEP	O
]	O
v	O
n	O

The	O
above	O
two	O
definitions	O
of	O
f	O
use	O
the	O
values	O
v	O
i	O
,	O
but	O
not	O
the	O
attributes	O
k	O
i	O
,	O
which	O
also	O
contain	O
meaningful	O
information	O
.	O
For	O
example	O
,	O
if	O
an	O
entity	O
seen	O
during	O
inference	O
has	O
a	O
capital	O
attribute	O
with	O
the	O
value	O
"	O
New	O
Delhi	O
"	O
,	O
seeing	O
the	O
capital	O
attribute	O
allows	O
us	O
to	O
infer	O
that	O
the	O
target	O
entity	O
is	O
likely	O
to	O
be	O
a	O
place	O
,	O
rather	O
than	O
a	O
person	O
,	O
especially	O
if	O
we	O
have	O
seen	O
the	O
capital	O
attribute	O
during	O
training	O
.	O
We	O
capture	O
this	O
information	O
using	O
attribute	O
separators	O
,	O
which	O
are	O
reserved	O
tokens	O
(	O
in	O
the	O
vein	O
of	O
[	O
SEP	O
]	O
tokens	O
)	O
corresponding	O
to	O
attributes	O
.	O
In	O
this	O
case	O
,	O

f	O
(	O
{	O
k	O
i	O
,	O
v	O
i	O
}	O
n	O
i=1	O
)	O
=	O
[	O
K	O
1	O
]	O
v	O
1	O
[	O
K	O
2	O
]	O
v	O
2	O
...	O
[	O
K	O
n	O
]	O
v	O
n	O
.	O

These	O
Figure	O
1	O
illustrates	O
the	O
three	O
instantiations	O
of	O
f	O
.	O
In	O
all	O
cases	O
,	O
attribute	O
-	O
value	O
pairs	O
are	O
ordered	O
in	O
descending	O
order	O
of	O
the	O
frequency	O
with	O
which	O
they	O
appear	O
in	O
the	O
training	O
KB	O
.	O
Finally	O
,	O
since	O
both	O
the	O
candidate	O
generation	O
and	O
candidate	O
re	O
-	O
ranking	O
models	O
we	O
build	O
on	O
use	O
BERT	O
,	O
the	O
techniques	O
discussed	O
here	O
can	O
be	O
applied	O
to	O
both	O
stages	O
,	O
but	O
we	O
only	O
focus	O
on	O
re	O
-	O
ranking	O
.	O

Regularization	O
Schemes	O
for	O
Improving	O
Generalization	O

Building	O
models	O
for	O
entity	O
linking	O
against	O
unseen	O
KBs	O
requires	O
that	O
such	O
models	O
do	O
not	O
overfit	O
to	O
the	O
training	O
data	O
by	O
memorizing	O
characteristics	O
of	O
the	O
training	O
KB	O
.	O
This	O
is	O
done	O
by	O
using	O
two	O
regularization	O
schemes	O
that	O
we	O
apply	O
on	O
top	O
of	O
the	O
candidate	O
string	O
generation	O
techniques	O
discussed	O
in	O
the	O
previous	O
section	O
.	O

The	O
first	O
scheme	O
,	O
which	O
we	O
call	O
attribute	O
-	O
OOV	O
,	O
prevents	O
models	O
from	O
overtly	O
relying	O
on	O
individual	O
[	O
K	O
i	O
]	O
tokens	O
and	O
generalize	O
to	O
attributes	O
that	O
are	O
not	O
seen	O
during	O
training	O
.	O
Analogous	O
to	O
how	O
out	O
-	O
of	O
-	O
vocabulary	O
tokens	O
are	O
commonly	O
handled	O
(	O
Dyer	O
et	O
al	O
.	O
,	O
2015	O
,	O
inter	O
alia	O
)	O
,	O
every	O
[	O
K	O
i	O
]	O
token	O
is	O
stochastically	O
replaced	O
with	O
the	O
[	O
SEP	O
]	O
token	O
during	O
training	O
with	O
probability	O
p	O
drop	O
.	O
This	O
encourages	O
the	O
model	O
to	O
encode	O
semantics	O
of	O
the	O
attributes	O
in	O
not	O
only	O
the	O
[	O
K	O
i	O
]	O
tokens	O
,	O
but	O
also	O
in	O
the	O
[	O
SEP	O
]	O
token	O
,	O
which	O
is	O
used	O
when	O
unseen	O
attributes	O
are	O
encountered	O
during	O
inference	O
.	O

The	O
second	O
regularization	O
scheme	O
discourages	O
the	O
model	O
from	O
memorizing	O
the	O
order	O
in	O
which	O
particular	O
attributes	O
occur	O
.	O
Under	O
attribute	O
-	O
shuffle	O
,	O
every	O
time	O
an	O
entity	O
is	O
encountered	O
during	O
training	O
,	O
its	O
attribute	O
/	O
values	O
are	O
randomly	O
shuffled	O
before	O
it	O
is	O
converted	O
to	O
a	O
string	O
representation	O
using	O
the	O
techniques	O
from	O
Section	O
4.1	O
.	O

Experiments	O
and	O
Discussion	O

Data	O

Our	O
held	O
-	O
out	O
test	O
bed	O
is	O
the	O
TAC	O
-	O
KBP	O
2010	O
data	O
(	O
LDC2018T16	O
)	O
which	O
consists	O
of	O
documents	O
from	O
English	O
newswire	O
,	O
discussion	O
forum	O
and	O
web	O
data	O
(	O
Ji	O
et	O
al	O
.	O
,	O
2010	O
)	O
.	O
4	O
The	O
target	O
KB	O
(	O
KB	O
test	O
)	O
is	O
the	O
TAC	O
-	O
KBP	O
Reference	O
KB	O
and	O
is	O
built	O
from	O
English	O
Wikipedia	O
articles	O
and	O
their	O
associated	O
infoboxes	O
(	O
LDC2014T16	O
)	O
.	O
5	O
Our	O
primary	O
training	O
and	O
validation	O
data	O
is	O
the	O
CoNLL	O
-	O
YAGO	O
dataset	O
(	O
Hoffart	O
et	O
al	O
.	O
,	O
2011	O
)	O
Table	O
2	O
describes	O
the	O
sizes	O
of	O
these	O
various	O
datasets	O
along	O
with	O
the	O
number	O
of	O
entities	O
in	O
their	O
respective	O
KBs	O
.	O

While	O
covering	O
similar	O
domains	O
,	O
Wikidata	O
and	O
the	O
TAC	O
-	O
KBP	O
Reference	O
KB	O
have	O
different	O
schemas	O
.	O
Wikidata	O
is	O
more	O
structured	O
and	O
entities	O
are	O
associated	O
with	O
statements	O
represented	O
using	O
attribute	O
-	O
value	O
pairs	O
,	O
which	O
are	O
short	O
snippets	O
rather	O
than	O
full	O
sentences	O
.	O
The	O
TAC	O
-	O
KBP	O
Reference	O
KB	O
contains	O
both	O
short	O
snippets	O
like	O
these	O
,	O
along	O
with	O
the	O
text	O
of	O
the	O
Wikipedia	O
article	O
of	O
the	O
entity	O
.	O
The	O
two	O
KBs	O
also	O
differ	O
in	O
size	O
,	O
with	O
Wikidata	O
containing	O
almost	O
seven	O
times	O
the	O
number	O
of	O
entities	O
in	O
TAC	O
KBP	O
.	O

Both	O
during	O
training	O
and	O
inference	O
,	O
we	O
only	O
retain	O
the	O
100	O
most	O
frequent	O
attributes	O
in	O
the	O
respective	O
KBs	O
.	O
The	O
attribute	O
-	O
separators	O
(	O
Section	O
4.1	O
)	O
are	O
created	O
corresponding	O
to	O
the	O
100	O
most	O
frequent	O
attributes	O
in	O
the	O
training	O
KB	O
.	O
Candidates	O
and	O
mentions	O
(	O
with	O
context	O
)	O
are	O
represented	O
using	O
strings	O
of	O
128	O
sub	O
-	O
word	O
tokens	O
each	O
,	O
across	O
all	O
models	O
.	O

839	O

Hyperparameters	O

All	O
BERT	O
models	O
are	O
uncased	O
BERT	O
-	O
base	O
models	O
with	O
12	O
layers	O
,	O
768	O
hidden	O
units	O
,	O
and	O
12	O
heads	O
with	O
default	O
parameters	O
,	O
and	O
trained	O
on	O
English	O
Wikipedia	O
and	O
the	O
BookCorpus	O
.	O
The	O
probability	O
p	O
drop	O
for	O
attribute	O
-	O
OOV	O
is	O
set	O
to	O
0.3	O
.	O
Both	O
candidate	O
generation	O
and	O
re	O
-	O
ranking	O
models	O
are	O
trained	O
using	O
the	O
BERT	O
Adam	O
optimizer	O
(	O
Kingma	O
and	O
Ba	O
,	O
2015	O
)	O
,	O
with	O
a	O
linear	O
warmup	O
for	O
10	O
%	O
of	O
the	O
first	O
epoch	O
to	O
a	O
peak	O
learning	O
rate	O
of	O
2	O
×	O
10	O
−5	O
and	O
a	O
linear	O
decay	O
from	O
there	O
till	O
the	O
learning	O
rate	O
approaches	O
zero	O
.	O
9	O
Candidate	O
generation	O
models	O
are	O
trained	O
for	O
200	O
epochs	O
with	O
a	O
batch	O
size	O
of	O
256	O
.	O
Re	O
-	O
ranking	O
models	O
are	O
trained	O
for	O
4	O
epochs	O
with	O
a	O
batch	O
size	O
of	O
2	O
,	O
and	O
operate	O
on	O
the	O
top	O
32	O
candidates	O
returned	O
by	O
the	O
generation	O
model	O
.	O
Hyperparameters	O
are	O
chosen	O
such	O
that	O
models	O
can	O
be	O
run	O
on	O
a	O
single	O
NVIDIA	O
V100	O
Tensor	O
Core	O
GPU	O
with	O
32	O
GB	O
RAM	O
,	O
and	O
are	O
not	O
extensively	O
tuned	O
.	O
All	O
models	O
have	O
the	O
same	O
number	O
of	O
parameters	O
except	O
the	O
ones	O
with	O
attribute	O
-	O
separators	O
which	O
have	O
100	O
extra	O
token	O
embeddings	O
(	O
of	O
size	O
768	O
each	O
)	O
.	O

Candidate	O
generation	O
Since	O
the	O
focus	O
of	O
our	O
experiments	O
is	O
on	O
re	O
-	O
ranking	O
,	O
we	O
use	O
a	O
fixed	O
candidate	O
generation	O
model	O
for	O
all	O
experiments	O
that	O
combines	O
the	O
architecture	O
of	O
Wu	O
et	O
al	O
.	O
(	O
2020	O
)	O
(	O
Section	O
3	O
)	O
with	O
[	O
SEP]-separation	O
to	O
generate	O
candidate	O
strings	O
.	O
This	O
model	O
also	O
has	O
no	O
knowledge	O
of	O
the	O
test	O
KB	O
and	O
is	O
trained	O
only	O
once	O
on	O
the	O
CoNLL	O
-	O
Wikidata	O
dataset	O
.	O
It	O
achieves	O
a	O
recall@32	O
of	O
91.25	O
when	O
evaluated	O
on	O
the	O
unseen	O
TAC	O
-	O
KBP	O
2010	O
data	O
.	O

Research	O
Questions	O

We	O
evaluate	O
the	O
re	O
-	O
ranking	O
model	O
(	O
Section	O
3	O
)	O
in	O
several	O
settings	O
to	O
answer	O
the	O
following	O
questions	O
:	O
For	O
all	O
experiments	O
,	O
we	O
report	O
the	O
mean	O
and	O
standard	O
deviation	O
of	O
the	O
accuracy	O
across	O
five	O
runs	O
with	O
different	O
random	O
seeds	O
.	O

Main	O
results	O

Our	O
primary	O
experiments	O
focus	O
on	O
the	O
first	O
two	O
research	O
questions	O
and	O
study	O
the	O
accuracy	O
of	O
the	O
model	O
that	O
uses	O
the	O
re	O
-	O
ranking	O
architecture	O
from	O
Section	O
3	O
with	O
the	O
three	O
core	O
components	O
introduced	O
in	O
Section	O
4	O
viz	O
.	O
attribute	O
-	O
separators	O
to	O
generate	O
string	O
representations	O
of	O
candidates	O
,	O
along	O
with	O
attribute	O
-	O
OOV	O
and	O
attribute	O
-	O
shuffle	O
for	O
regularization	O
.	O
We	O
compare	O
this	O
against	O
two	O
baselines	O
without	O
these	O
components	O
that	O
use	O
the	O
same	O
architecture	O
and	O
use	O
concatenation	O
and	O
[	O
SEP]separation	O
instead	O
of	O
attribute	O
-	O
separators	O
.	O
As	O
a	O
reminder	O
,	O
all	O
models	O
are	O
trained	O
as	O
well	O
as	O
validated	O
on	O
CoNLL	O
-	O
Wikidata	O
and	O
evaluated	O
on	O
the	O
completely	O
unseen	O
TAC	O
-	O
KBP	O
2010	O
test	O
set	O
.	O

Results	O
confirm	O
that	O
adding	O
structure	O
to	O
the	O
candidate	O
string	O
representations	O
via	O
[	O
SEP	O
]	O
tokens	O
leads	O
to	O
more	O
accurate	O
models	O
compared	O
to	O
generating	O
strings	O
by	O
concatenation	O
(	O
Table	O
3	O
)	O
.	O
Using	O
attributeseparators	O
instead	O
of	O
[	O
SEP	O
]	O
tokens	O
leads	O
to	O
an	O
absolute	O
gain	O
of	O
over	O
5	O
%	O
and	O
handling	O
unseen	O
attributes	O
via	O
attribute	O
-	O
OOV	O
further	O
increases	O
the	O
accuracy	O
to	O
56.2	O
%	O
,	O
a	O
7.1	O
%	O
increase	O
over	O
the	O
[	O
SEP	O
]	O
baseline	O
.	O
These	O
results	O
show	O
that	O
the	O
attributeseparators	O
capture	O
meaningful	O
information	O
about	O
attributes	O
,	O
even	O
when	O
only	O
a	O
small	O
number	O
of	O
attributes	O
from	O
the	O
training	O
data	O
(	O
15	O
)	O
are	O
observed	O
during	O
inference	O
.	O
Shuffling	O
attribute	O
-	O
value	O
pairs	O
before	O
converting	O
them	O
to	O
a	O
string	O
representation	O
using	O
attributeseparators	O
also	O
independently	O
provides	O
an	O
absolute	O
gain	O
of	O
3.5	O
%	O
over	O
the	O
model	O
which	O
uses	O
attribute	O
-	O
separators	O
without	O
shuffling	O
.	O
Overall	O
,	O
models	O
that	O
combine	O
attribute	O
-	O
shuffling	O
and	O
attribute	O
-	O
OOV	O
are	O
the	O
most	O
accurate	O
with	O
an	O
accuracy	O
of	O
61.6	O
%	O
,	O
which	O
represents	O
a	O
12	O
%	O
absolute	O
gain	O
over	O
the	O
best	O
baseline	O
model	O
.	O

Prior	O
work	O
(	O
Raiman	O
and	O
Raiman	O
,	O
2018;Cao	O
et	O
al	O
.	O
,	O
2018;Wu	O
et	O
al	O
.	O
,	O
2020;Févry	O
et	O
al	O
.	O
,	O
2020	O
)	O
reports	O
higher	O
accuracies	O
on	O
the	O
TAC	O
data	O
but	O
they	O
are	O
fundamentally	O
incomparable	O
with	O
our	O
numbers	O
due	O
to	O
the	O
simple	O
fact	O
that	O
we	O
are	O
solving	O
a	O
different	O
task	O
with	O
three	O
key	O
differences	O
:	O
(	O
1	O
)	O
Models	O
in	O
prior	O
work	O
are	O
trained	O
and	O
evaluated	O
using	O
mentions	O
that	O
link	O
to	O
the	O
same	O
KB	O
.	O
On	O
the	O
contrary	O
,	O
we	O
show	O
how	O
far	O
we	O
can	O
go	O
without	O
such	O
in	O
-	O
KB	O
training	O
mentions	O
.	O

(	O
2	O
)	O
The	O
test	O
KB	O
used	O
by	O
these	O
works	O
is	O
different	O
from	O
our	O
test	O
KB	O
.	O
Each	O
entry	O
in	O
the	O
KB	O
used	O
by	O
prior	O
work	O
simply	O
consists	O
of	O
the	O
name	O
of	O
the	O
entity	O
with	O
a	O
textual	O
description	O
,	O
while	O
each	O
entity	O
in	O
our	O
KB	O
is	O
represented	O
via	O
multiple	O
attribute	O
-	O
value	O
pairs	O
.	O
(	O
3	O
)	O
These	O
models	O
exploit	O
the	O
homogeneous	O
nature	O
of	O
the	O
KBs	O
and	O
usually	O
pre	O
-	O
train	O
models	O
on	O
millions	O
of	O
mentions	O
from	O
Wikipedia	O
.	O
This	O
is	O
beneficial	O
when	O
the	O
training	O
and	O
test	O
KBs	O
are	O
Wikipedia	O
or	O
similar	O
,	O
but	O
is	O
beyond	O
the	O
scope	O
of	O
this	O
work	O
,	O
as	O
we	O
build	O
models	O
applicable	O
to	O
arbitrary	O
databases	O
.	O

Training	O
on	O
multiple	O
unrelated	O
datasets	O

An	O
additional	O
benefit	O
of	O
being	O
able	O
to	O
link	O
to	O
multiple	O
KBs	O
is	O
the	O
ability	O
to	O
train	O
on	O
more	O
than	O
one	O
dataset	O
,	O
each	O
of	O
which	O
links	O
to	O
a	O
different	O
KB	O
with	O
different	O
schemas	O
.	O
While	O
prior	O
work	O
has	O
been	O
unable	O
to	O
do	O
so	O
due	O
to	O
its	O
reliance	O
on	O
knowledge	O
of	O
KB	O
test	O
,	O
this	O
ability	O
is	O
more	O
crucial	O
in	O
the	O
settings	O
we	O
investigate	O
,	O
as	O
it	O
allows	O
us	O
to	O
stack	O
independent	O
datasets	O
for	O
training	O
.	O
This	O
allows	O
us	O
to	O
answer	O
our	O
third	O
research	O
question	O
.	O
Specifically	O
,	O
we	O
compare	O
the	O
[	O
SEP]-separation	O
baseline	O
with	O
our	O
full	O
model	O
that	O
uses	O
attribute	O
-	O
separators	O
,	O
attributeshuffle	O
,	O
and	O
attribute	O
-	O
OOV	O
.	O
We	O
ask	O
whether	O
the	O
%	O
of	O
TAC	O
4	O
)	O
.	O
In	O
contrast	O
,	O
the	O
baseline	O
model	O
observes	O
a	O
bigger	O
increase	O
in	O
accuracy	O
from	O
49.1	O
%	O
to	O
62.6	O
%	O
.	O
While	O
the	O
difference	O
between	O
the	O
two	O
models	O
reduces	O
,	O
the	O
full	O
model	O
remains	O
more	O
accurate	O
.	O
These	O
results	O
also	O
show	O
that	O
the	O
seamless	O
stacking	O
of	O
multiple	O
datasets	O
allowed	O
by	O
our	O
models	O
is	O
effective	O
empirically	O
.	O

Impact	O
of	O
schema	O
-	O
aware	O
training	O
data	O

Finally	O
,	O
we	O
investigate	O
to	O
what	O
extent	O
do	O
components	O
introduced	O
by	O
us	O
help	O
in	O
linking	O
when	O
there	O
is	O
training	O
data	O
available	O
that	O
links	O
to	O
the	O
inference	O
KB	O
,	O
KB	O
test	O
.	O
We	O
hypothesize	O
that	O
while	O
attributeseparators	O
will	O
still	O
be	O
useful	O
,	O
attribute	O
-	O
OOV	O
and	O
attribute	O
-	O
shuffle	O
will	O
be	O
less	O
useful	O
as	O
there	O
is	O
a	O
smaller	O
gap	O
between	O
training	O
and	O
test	O
scenarios	O
,	O
reducing	O
the	O
need	O
for	O
regularization	O
.	O

For	O
these	O
experiments	O
,	O
models	O
from	O
Section	O
5.4	O
are	O
further	O
trained	O
with	O
increasing	O
amounts	O
of	O
data	O
from	O
the	O
TAC	O
-	O
KBP	O
2010	O
training	O
set	O
.	O
A	O
sample	O
of	O
200	O
documents	O
is	O
held	O
out	O
from	O
the	O
training	O
data	O
as	O
a	O
validation	O
set	O
.	O
The	O
models	O
are	O
trained	O
with	O
the	O
exact	O
same	O
configuration	O
as	O
the	O
base	O
models	O
,	O
except	O
with	O
a	O
smaller	O
constant	O
learning	O
rate	O
of	O
2	O
×	O
10	O
−6	O
to	O
not	O
overfit	O
on	O
the	O
small	O
amounts	O
of	O
data	O
.	O
Unsurprisingly	O
,	O
the	O
accuracy	O
of	O
all	O
models	O
increases	O
as	O
the	O
amount	O
of	O
TAC	O
training	O
data	O
in-	O
Crucially	O
,	O
the	O
model	O
with	O
only	O
attribute	O
separators	O
is	O
the	O
most	O
accurate	O
model	O
across	O
the	O
spectrum	O
.	O
Moreover	O
,	O
the	O
difference	O
between	O
this	O
model	O
and	O
the	O
baseline	O
model	O
sharply	O
increases	O
as	O
the	O
amount	O
of	O
schema	O
-	O
aware	O
data	O
decreases	O
(	O
e.g.	O
when	O
using	O
13	O
annotated	O
documents	O
,	O
i.e.	O
1	O
%	O
of	O
the	O
training	O
data	O
,	O
we	O
get	O
a	O
9	O
%	O
boost	O
in	O
accuracy	O
over	O
the	O
model	O
that	O
does	O
not	O
see	O
any	O
schema	O
-	O
aware	O
data	O
)	O
.	O
These	O
trends	O
show	O
that	O
our	O
models	O
are	O
not	O
only	O
useful	O
in	O
settings	O
without	O
any	O
data	O
from	O
the	O
target	O
KB	O
,	O
but	O
also	O
in	O
settings	O
where	O
limited	O
data	O
is	O
available	O
.	O

Qualitative	O
Analysis	O

Beyond	O
the	O
quantitative	O
evaluations	O
above	O
,	O
we	O
further	O
qualitatively	O
analyze	O
the	O
predictions	O
of	O
the	O
best	O
model	O
from	O
Table	O
3	O
to	O
provide	O
insights	O
into	O
our	O
modeling	O
decisions	O
and	O
suggest	O
avenues	O
for	O
improvements	O
.	O

Improvements	O
over	O
baseline	O

First	O
,	O
we	O
categorize	O
all	O
newly	O
correct	O
mentions	O
,	O
i.e.	O
mentions	O
that	O
are	O
correctly	O
linked	O
by	O
the	O
top	O
model	O
but	O
incorrectly	O
linked	O
by	O
the	O
[	O
SEP]-separation	O
baseline	O
by	O
the	O
entity	O
type	O
of	O
the	O
gold	O
entity	O
.	O
This	O
type	O
is	O
one	O
of	O
person	O
(	O
PER	O
)	O
,	O
organization	O
(	O
ORG	O
)	O
,	O
geo	O
-	O
political	O
entity	O
(	O
GPE	O
)	O
,	O
and	O
a	O
catchall	O
unknown	O
10	O
The	O
0	O
%	O
results	O
are	O
the	O
same	O
as	O
those	O
in	O
Table	O
3	O
.	O
category	O
(	O
UKN	O
)	O
.	O
11	O
This	O
categorization	O
reveals	O
that	O
the	O
newly	O
correct	O
mentions	O
represent	O
about	O
15	O
%	O
of	O
the	O
total	O
mentions	O
of	O
the	O
ORG	O
,	O
GPE	O
,	O
and	O
UKN	O
categories	O
and	O
as	O
much	O
as	O
25	O
%	O
of	O
the	O
total	O
mentions	O
of	O
the	O
PER	O
category	O
.	O
This	O
distributed	O
improvement	O
highlights	O
that	O
the	O
relatively	O
higher	O
accuracy	O
of	O
our	O
model	O
is	O
due	O
to	O
a	O
holistic	O
improvement	O
in	O
modeling	O
unseen	O
KBs	O
across	O
all	O
entity	O
types	O
.	O

Why	O
does	O
PER	O
benefit	O
more	O
than	O
other	O
entity	O
types	O
?	O
To	O
answer	O
this	O
,	O
we	O
count	O
the	O
fraction	O
of	O
mentions	O
of	O
each	O
entity	O
type	O
that	O
have	O
at	O
least	O
one	O
column	O
represented	O
using	O
attribute	O
separators	O
.	O
This	O
counting	O
reveals	O
that	O
approximately	O
56	O
-	O
58	O
%	O
of	O
mentions	O
of	O
type	O
ORG	O
,	O
GPE	O
,	O
and	O
UKN	O
have	O
at	O
least	O
one	O
such	O
column	O
.	O
On	O
the	O
other	O
hand	O
,	O
this	O
number	O
is	O
71	O
%	O
for	O
PER	O
mentions	O
.	O
This	O
suggests	O
that	O
the	O
difference	O
is	O
directly	O
attributable	O
to	O
more	O
PER	O
entities	O
having	O
a	O
column	O
that	O
has	O
been	O
modeled	O
using	O
attribute	O
separators	O
,	O
further	O
highlighting	O
the	O
benefits	O
of	O
this	O
modeling	O
decision	O
.	O

Error	O
Analysis	O

To	O
identify	O
the	O
shortcomings	O
of	O
our	O
best	O
model	O
,	O
we	O
categorize	O
100	O
random	O
mentions	O
that	O
are	O
incorrectly	O
linked	O
by	O
this	O
model	O
into	O
six	O
categories	O
(	O
demonstrated	O
with	O
examples	O
in	O
Table	O
6	O
)	O
,	O
inspired	O
by	O
the	O
taxonomy	O
of	O
.	O

Under	O
this	O
taxonomy	O
,	O
a	O
common	O
error	O
(	O
33	O
%	O
)	O
is	O
predicting	O
a	O
more	O
specific	O
entity	O
than	O
that	O
indicated	O
by	O
the	O
mention	O
(	O
the	O
city	O
of	O
Hartford	O
,	O
Connecticut	O
,	O
rather	O
than	O
the	O
state	O
)	O
.	O
The	O
reverse	O
is	O
also	O
observed	O
(	O
i.e.	O
the	O
model	O
predicts	O
a	O
more	O
general	O
entity	O
)	O
,	O
but	O
far	O
less	O
frequently	O
(	O
6	O
%	O
)	O
.	O
Another	O
major	O
error	O
category	O
(	O
33	O
%	O
)	O
is	O
when	O
the	O
model	O
fails	O
to	O
pick	O
up	O
the	O
correct	O
signals	O
from	O
the	O
context	O
and	O
assigns	O
a	O
similarly	O
named	O
entity	O
of	O
a	O
similar	O
type	O
(	O
e.g.	O
the	O
river	O
Mobile	O
,	O
instead	O
of	O
the	O
city	O
Mobile	O
,	O
both	O
of	O
which	O
are	O
locations	O
)	O
.	O
21	O
%	O
of	O
the	O
errors	O
are	O
cases	O
where	O
the	O
model	O
predicts	O
an	O
entity	O
that	O
is	O
related	O
to	O
the	O
gold	O
entity	O
,	O
but	O
is	O
neither	O
more	O
specific	O
,	O
nor	O
more	O
generic	O
,	O
but	O
rather	O
of	O
a	O
different	O
type	O
(	O
Santos	O
Football	O
Club	O
instead	O
of	O
the	O
city	O
of	O
Santos	O
)	O
.	O

Errors	O
in	O
the	O
last	O
category	O
occur	O
when	O
the	O
model	O
predicts	O
an	O
entity	O
whose	O
name	O
has	O
no	O
string	O
overlap	O
with	O
that	O
of	O
the	O
gold	O
entity	O
or	O
the	O
mention	O
.	O
This	O
likely	O
happens	O
when	O
the	O
signals	O
from	O
the	O
context	O
override	O
the	O
signals	O
from	O
the	O
mention	O
itself	O
.	O

Conclusion	O

The	O
primary	O
contribution	O
of	O
this	O
work	O
is	O
a	O
novel	O
framework	O
for	O
entity	O
linking	O
against	O
unseen	O
target	O
KBs	O
with	O
unknown	O
schemas	O
.	O
To	O
this	O
end	O
,	O
we	O
introduce	O
methods	O
to	O
generalize	O
existing	O
models	O
for	O
zero	O
-	O
shot	O
entity	O
linking	O
to	O
link	O
to	O
unseen	O
KBs	O
.	O
These	O
methods	O
rely	O
on	O
converting	O
arbitrary	O
entities	O
represented	O
using	O
a	O
set	O
of	O
attribute	O
-	O
value	O
pairs	O
into	O
a	O
string	O
representation	O
that	O
can	O
be	O
then	O
consumed	O
by	O
models	O
from	O
prior	O
work	O
.	O

There	O
is	O
still	O
a	O
significant	O
gap	O
between	O
models	O
used	O
in	O
this	O
work	O
and	O
schema	O
-	O
aware	O
models	O
that	O
are	O
trained	O
on	O
the	O
same	O
KB	O
as	O
the	O
inference	O
KB	O
.	O
One	O
way	O
to	O
close	O
this	O
gap	O
is	O
by	O
using	O
automatic	O
table	O
-	O
to	O
-	O
text	O
generation	O
techniques	O
to	O
convert	O
arbitrary	O
entities	O
into	O
fluent	O
and	O
adequate	O
text	O
(	O
Kukich	O
,	O
1983;McKeown	O
,	O
1985;Reiter	O
and	O
Dale	O
,	O
1997;Wiseman	O
et	O
al	O
.	O
,	O
2017;Chisholm	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
Another	O
promising	O
direction	O
is	O
to	O
move	O
beyond	O
BERT	O
to	O
other	O
pre	O
-	O
trained	O
representations	O
that	O
are	O
better	O
known	O
to	O
encode	O
entity	O
information	O
(	O
Zhang	O
et	O
al	O
.	O
,	O
2019;Guu	O
et	O
al	O
.	O
,	O
2020;Poerner	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

Finally	O
,	O
while	O
the	O
focus	O
of	O
this	O
work	O
is	O
only	O
on	O
English	O
entity	O
linking	O
,	O
challenges	O
associated	O
with	O
this	O
work	O
naturally	O
occur	O
in	O
multilingual	O
settings	O
as	O
well	O
.	O
Just	O
as	O
we	O
can	O
not	O
expect	O
labeled	O
data	O
for	O
every	O
target	O
KB	O
of	O
interest	O
,	O
we	O
also	O
can	O
not	O
expect	O
labeled	O
data	O
for	O
different	O
KBs	O
in	O
different	O
languages	O
.	O
In	O
future	O
work	O
,	O
we	O
aim	O
to	O
investigate	O
how	O
we	O
can	O
port	O
the	O
solutions	O
introduced	O
here	O
to	O
multilingual	O
settings	O
as	O
well	O
develop	O
novel	O
solutions	O
for	O
scenarios	O
where	O
the	O
documents	O
and	O
the	O
KB	O
are	O
in	O
languages	O
other	O
than	O
English	O
(	O
Sil	O
et	O
al	O
.	O
,	O
2018;Upadhyay	O
et	O
al	O
.	O
,	O
2018;Botha	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

Acknowledgements	O

The	O
authors	O
would	O
like	O
to	O
thank	O
colleagues	O
from	O
Amazon	O
AI	O
for	O
many	O
helpful	O
discussions	O
that	O
shaped	O
this	O
work	O
,	O
and	O
for	O
reading	O
and	O
providing	O
feedback	O
on	O
earlier	O
drafts	O
of	O
the	O
paper	O
.	O
They	O
also	O
thank	O
all	O
the	O
anonymous	O
reviewers	O
for	O
their	O
helpful	O
feedback	O
.	O

Datasets	O
.	O
We	O
conduct	O
experiments	O
on	O
English	O
-	O
Macedonian	O
(	O
En	O
-	O
Mk	O
)	O
and	O
English	O
-	O
Albanian	O
(	O
En	O
-	O
Sq	O
)	O
,	O
as	O
Mk	O
,	O
Sq	O
are	O
low	O
-	O
resource	O
languages	O
,	O
where	O
lexical	O
-	O
level	O
alignment	O
can	O
be	O
most	O
beneficial	O
.	O
We	O
use	O
3	O
K	O
randomly	O
sampled	O
sentences	O
of	O
SETIMES	O
(	O
Tiedemann	O
,	O
2012	O
)	O
as	O
validation	O
/	O
test	O
sets	O
.	O
We	O
also	O
use	O
68	O
M	O
En	O
sentences	O
from	O
NewsCrawl	O
.	O
For	O
Sq	O
and	O
Mk	O
we	O
use	O
all	O
the	O
CommonCrawl	O
corpora	O
from	O
Ortiz	O
Suárez	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
which	O
are	O
4	O
M	O
Sq	O
and	O
2.4	O
M	O
Mk	O
sentences	O
.	O

Baseline	O
.	O
We	O
use	O
a	O
method	O
that	O
relies	O
on	O
crosslingual	O
language	O
model	O
pretraining	O
,	O
namely	O
XLM	O
(	O
Lample	O
and	O
Conneau	O
,	O
2019	O
)	O
.	O
This	O
approach	O
trains	O
a	O
bilingual	O
MLM	O
separately	O
for	O
En	O
-	O
Mk	O
and	O
En	O
-	O
Sq	O
,	O
which	O
is	O
used	O
to	O
initialize	O
the	O
encoder	O
-	O
decoder	O
of	O
the	O
corresponding	O
NMT	O
system	O
.	O
Each	O
system	O
is	O
then	O
trained	O
in	O
an	O
unsupervised	O
way	O
.	O

The	O
scores	O
presented	O
are	O
significantly	O
different	O
(	O
p	O
<	O
0.05	O
)	O
from	O
the	O
respective	O
baseline	O
.	O
CHRF1	O
refers	O
to	O
character	O
n	O
-	O
gram	O
F1	O
score	O
(	O
Popović	O
,	O
2015	O
)	O
.	O
The	O
models	O
in	O
italics	O
are	O
ours	O
.	O

Table	O
1	O
shows	O
the	O
results	O
of	O
our	O
approach	O
compared	O
to	O
two	O
pretraining	O
approaches	O
that	O
rely	O
on	O
In	O
the	O
case	O
of	O
XLM	O
,	O
the	O
effect	O
of	O
cross	O
-	O
lingual	O
lexical	O
alignment	O
is	O
more	O
evident	O
for	O
En	O
-	O
Mk	O
,	O
as	O
Mk	O
is	O
less	O
similar	O
to	O
En	O
,	O
compared	O
to	O
Sq	O
.	O
This	O
is	O
mainly	O
the	O
case	O
because	O
the	O
two	O
languages	O
use	O
a	O
different	O
alphabet	O
(	O
Latin	O
for	O
En	O
and	O
Cyrillic	O
for	O
Mk	O
)	O
.	O
This	O
is	O
also	O
true	O
for	O
RE	O
-	O
LM	O
when	O
translating	O
out	O
of	O
En	O
,	O
showing	O
that	O
enhancing	O
the	O
fine	O
-	O
tuning	O
step	O
of	O
MLM	O
with	O
pretrained	O
embeddings	O
is	O
helpful	O
and	O
improves	O
the	O
final	O
UNMT	O
performance	O
.	O

In	O
Table	O
2	O
,	O
we	O
observe	O
that	O
lexical	O
alignment	O
is	O
more	O
beneficial	O
for	O
En	O
-	O
Mk	O
.	O
This	O
can	O
be	O
explained	O
by	O
the	O
limited	O
vocabulary	O
overlap	O
of	O
the	O
two	O
languages	O
,	O
which	O
does	O
not	O
provide	O
sufficient	O
crosslingual	O
signal	O
for	O
the	O
training	O
of	O
MLM	O
.	O
By	O
contrast	O
,	O
initializing	O
an	O
MLM	O
with	O
pretrained	O
embeddings	O
largely	O
improves	O
performance	O
,	O
even	O
for	O
a	O
higherperforming	O
model	O
,	O
such	O
as	O
RE	O
-	O
LM	O
.	O
In	O
En	O
-	O
Sq	O
,	O
the	O
effect	O
of	O
our	O
approach	O
is	O
smaller	O
yet	O
consistent	O
.	O
This	O
can	O
be	O
attributed	O
to	O
the	O
fact	O
that	O
the	O
two	O
languages	O
use	O
the	O
same	O
script	O
.	O

Overall	O
,	O
our	O
method	O
enhances	O
the	O
lexical	O
-	O
level	O
information	O
captured	O
by	O
pretrained	O
MLMs	O
,	O
as	O
shown	O
empirically	O
.	O
This	O
is	O
consistent	O
with	O
our	O
intuition	O
that	O
cross	O
-	O
lingual	O
embeddings	O
capture	O
a	O
bilingual	O
signal	O
that	O
can	O
benefit	O
MLM	O
representations	O
.	O
1	O
-	O
gram	O
precision	O
scores	O
.	O
To	O
examine	O
whether	O
the	O
improved	O
translation	O
performance	O
is	O
a	O
result	O
of	O
the	O
lexical	O
-	O
level	O
information	O
provided	O
by	O
static	O
embeddings	O
,	O
we	O
present	O
1	O
-	O
gram	O
precision	O
scores	O
in	O
Ta-	O
ble	O
3	O
,	O
as	O
they	O
can	O
be	O
directly	O
attributed	O
to	O
lexical	O
alignment	O
.	O
The	O
biggest	O
performance	O
gains	O
(	O
up	O
to	O
+10.4	O
)	O
are	O
obtained	O
when	O
the	O
proposed	O
approach	O
is	O
applied	O
to	O
XLM	O
.	O
This	O
correlates	O
with	O
the	O
BLEU	O
scores	O
of	O
Table	O
1	O
.	O
Moreover	O
,	O
the	O
En	O
-	O
Mk	O
language	O
pair	O
benefits	O
more	O
than	O
En	O
-	O
Sq	O
from	O
the	O
lexicallevel	O
alignment	O
both	O
in	O
terms	O
of	O
1	O
-	O
gram	O
precision	O
and	O
BLEU	O
.	O
These	O
results	O
show	O
that	O
the	O
improved	O
BLEU	O
scores	O
can	O
be	O
attributed	O
to	O
the	O
enhanced	O
lexical	O
representations	O
.	O
How	O
should	O
static	O
embeddings	O
be	O
integrated	O
in	O
the	O
MLM	O
training	O
?	O
We	O
explore	O
different	O
ways	O
of	O
incorporating	O
the	O
lexical	O
knowledge	O
of	O
pretrained	O
cross	O
-	O
lingual	O
embeddings	O
to	O
the	O
second	O
,	O
masked	O
language	O
modeling	O
stage	O
of	O
our	O
approach	O
(	O
§	O
2.2	O
)	O
.	O
Specifically	O
,	O
we	O
keep	O
the	O
aligned	O
embeddings	O
fixed	O
(	O
frozen	O
)	O
during	O
XLM	O
training	O
and	O
compare	O
the	O
performance	O
of	O
the	O
final	O
UNMT	O
model	O
to	O
the	O
proposed	O
(	O
fine	O
-	O
tuned	O
)	O
method	O
.	O
We	O
point	O
out	O
that	O
,	O
after	O
we	O
transfer	O
the	O
trained	O
MLM	O
to	O
an	O
encoder	O
-	O
decoder	O
model	O
,	O
all	O
layers	O
are	O
trained	O
for	O
UNMT	O
.	O

We	O
tie	O
the	O
embedding	O
and	O
output	O
(	O
projection	O
)	O
layers	O
of	O
both	O
LM	O
and	O
NMT	O
models	O
(	O
Press	O
and	O
Wolf	O
,	O
2017	O
)	O
.	O
We	O
use	O
a	O
dropout	O
rate	O
of	O
0.1	O
and	O
GELU	O
activations	O
(	O
Hendrycks	O
and	O
Gimpel	O
,	O
2017	O
)	O
.	O
We	O
use	O
the	O
default	O
parameters	O
of	O
Lample	O
and	O
Conneau	O
(	O
2019	O
)	O
in	O
order	O
to	O
train	O
our	O
models	O
.	O

In	O
this	O
work	O
,	O
we	O
focus	O
on	O
self	O
-	O
supervised	O
,	O
alignment	O
-	O
oriented	O
training	O
tasks	O
using	O
minimum	O
parallel	O
data	O
to	O
improve	O
mBERT	O
's	O
cross	O
-	O
lingual	O
transferability	O
.	O
We	O
propose	O
a	O
Post	O
-	O
Pretraining	O
Alignment	O
(	O
PPA	O
)	O
method	O
consisting	O
of	O
both	O
wordlevel	O
and	O
sentence	O
-	O
level	O
alignment	O
,	O
as	O
well	O
as	O
a	O
finetuning	O
technique	O
on	O
downstream	O
tasks	O
that	O
take	O
pairs	O
of	O
text	O
as	O
input	O
,	O
such	O
as	O
NLI	O
and	O
Question	O
Answering	O
(	O
QA	O
)	O
.	O
Specifically	O
,	O
we	O
use	O
a	O
slightly	O
different	O
version	O
of	O
TLM	O
as	O
our	O
word	O
-	O
level	O
alignment	O
task	O
and	O
contrastive	O
learning	O
(	O
Hadsell	O
et	O
al	O
.	O
,	O
2006	O
)	O
on	O
mBERT	O
's	O
[	O
CLS	O
]	O
tokens	O
to	O
align	O
sentence	O
-	O
level	O
representations	O
.	O
Both	O
tasks	O
are	O
self	O
-	O
supervised	O
and	O
do	O
not	O
require	O
pre	O
-	O
alignment	O
tools	O
such	O
as	O
FastAlign	O
.	O
Our	O
sentence	O
-	O
level	O
alignment	O
is	O
implemented	O
using	O
MoCo	O
(	O
He	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
an	O
instance	O
discrimination	O
-	O
based	O
method	O
of	O
contrastive	O
learn-	O
ing	O
that	O
was	O
recently	O
proposed	O
for	O
self	O
-	O
supervised	O
representation	O
learning	O
in	O
computer	O
vision	O
.	O
Lastly	O
,	O
when	O
finetuning	O
on	O
NLI	O
and	O
QA	O
tasks	O
for	O
non	O
-	O
English	O
languages	O
,	O
we	O
perform	O
sentence	O
-	O
level	O
codeswitching	O
with	O
English	O
as	O
a	O
form	O
of	O
both	O
alignment	O
and	O
data	O
augmentation	O
.	O
We	O
conduct	O
controlled	O
experiments	O
on	O
XNLI	O
and	O
MLQA	O
(	O
Lewis	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
leveraging	O
varying	O
amounts	O
of	O
parallel	O
data	O
during	O
alignment	O
.	O
We	O
then	O
conduct	O
an	O
ablation	O
study	O
that	O
shows	O
the	O
effectiveness	O
of	O
our	O
method	O
.	O
On	O
XNLI	O
,	O
our	O
aligned	O
mBERT	O
improves	O
over	O
the	O
original	O
mBERT	O
by	O
4.7	O
%	O
for	O
zero	O
-	O
shot	O
transfer	O
,	O
and	O
outperforms	O
Cao	O
et	O
al	O
.	O
(	O
2020	O
)	O
while	O
using	O
the	O
same	O
amount	O
of	O
parallel	O
data	O
from	O
the	O
same	O
source	O
.	O
For	O
translate	O
-	O
train	O
,	O
where	O
translation	O
of	O
English	O
training	O
data	O
is	O
available	O
in	O
the	O
target	O
language	O
,	O
our	O
model	O
achieves	O
comparable	O
performance	O
to	O
XLM	O
while	O
using	O
far	O
fewer	O
resources	O
.	O
On	O
MLQA	O
,	O
we	O
get	O
2.3	O
%	O
improvement	O
over	O
mBERT	O
and	O
outperform	O
XLM	O
-	O
R	O
Base	O
for	O
zero	O
-	O
shot	O
transfer	O
.	O

Concretely	O
,	O
MoCo	O
employs	O
a	O
dual	O
-	O
encoder	O
architecture	O
.	O
Given	O
two	O
views	O
v	O
1	O
and	O
v	O
2	O
of	O
the	O
same	O
image	O
,	O
v	O
1	O
is	O
encoded	O
by	O
the	O
query	O
encoder	O
f	O
q	O
and	O
v	O
2	O
by	O
the	O
momentum	O
encoder	O
f	O
k	O
.	O
v	O
1	O
and	O
v	O
2	O
form	O
a	O
positive	O
pair	O
.	O
Negative	O
examples	O
are	O
views	O
of	O
different	O
source	O
images	O
,	O
and	O
are	O
stored	O
in	O
a	O
queue	O
∈	O
K	O
,	O
which	O
is	O
randomly	O
initialized	O
.	O
K	O
is	O
usually	O
a	O
large	O
number	O
(	O
e.g.	O
,	O
K	O
=	O
65	O
,	O
536	O
for	O
ImageNet	O
)	O
.	O
Negative	O
pairs	O
are	O
formed	O
by	O
comparing	O
v	O
1	O
with	O
each	O
item	O
in	O
the	O
queue	O
.	O
Similarity	O
between	O
pairs	O
is	O
measured	O
by	O
dot	O
product	O
.	O
MoCo	O
uses	O
the	O
InfoNCE	O
loss	O
(	O
van	O
den	O
Oord	O
et	O
al	O
.	O
,	O
2019	O
)	O
to	O
bring	O
positive	O
pairs	O
closer	O
to	O
each	O
other	O
and	O
push	O
negative	O
pairs	O
apart	O
.	O
After	O
a	O
batch	O
of	O
view	O
pairs	O
are	O
processed	O
,	O
those	O
encoded	O
by	O
the	O
momentum	O
encoder	O
are	O
added	O
to	O
the	O
queue	O
as	O
negative	O
examples	O
for	O
future	O
queries	O
.	O
During	O
training	O
,	O
the	O
query	O
encoder	O
is	O
updated	O
by	O
the	O
optimizer	O
while	O
the	O
momentum	O
encoder	O
is	O
updated	O
by	O
the	O
exponential	O
moving	O
average	O
of	O
the	O
query	O
encoder	O
's	O
parameters	O
to	O
maintain	O
queue	O
consistency	O
:	O

For	O
both	O
PPA	O
and	O
finetuning	O
on	O
downstream	O
tasks	O
,	O
we	O
use	O
the	O
AdamW	O
optimizer	O
with	O
0.01	O
weight	O
decay	O
and	O
a	O
linear	O
learning	O
rate	O
scheduler	O
.	O
For	O
PPA	O
,	O
we	O
use	O
a	O
batch	O
size	O
of	O
128	O
,	O
mBERT	O
max	O
sequence	O
length	O
128	O
and	O
learning	O
rate	O
warmup	O
for	O
the	O
first	O
10	O
%	O
of	O
the	O
total	O
iterations	O
,	O
peaking	O
at	O
0.00003	O
.	O
The	O
MoCo	O
momentum	O
is	O
set	O
to	O
0.999	O
,	O
queue	O
size	O
32000	O
and	O
temperature	O
0.05	O
.	O
Our	O
PPA	O
models	O
are	O
trained	O
for	O
10	O
epochs	O
,	O
except	O
for	O
the	O
2	O
M	O
setting	O
where	O
5	O
epochs	O
are	O
trained	O
.	O
On	O
XNLI	O
,	O
we	O
use	O
a	O
batch	O
size	O
of	O
32	O
,	O
mBERT	O
max	O
sequence	O
length	O
128	O
and	O
finetune	O
the	O
PPA	O
model	O
for	O
2	O
epochs	O
.	O
Learning	O
rate	O
peaks	O
at	O
0.00005	O
and	O
warmup	O
is	O
done	O
to	O
the	O
first	O
1000	O
iterations	O
.	O
On	O
MLQA	O
,	O
mBERT	O
max	O
sequence	O
length	O
is	O
set	O
to	O
386	O
and	O
peak	O
learning	O
rate	O
0.00003	O
.	O
The	O
other	O
parameters	O
are	O
the	O
same	O
as	O
XNLI	O
.	O
Our	O
experiments	O
are	O
run	O
on	O
a	O
single	O
32	O
GB	O
V100	O
GPU	O
,	O
except	O
for	O
PPA	O
training	O
that	O
involves	O
either	O
MLM	O
or	O
TLM	O
,	O
where	O
two	O
such	O
GPUs	O
are	O
used	O
.	O
We	O
also	O
use	O
mixed	O
-	O
precision	O
training	O
to	O
save	O
on	O
GPU	O
memory	O
and	O
speed	O
up	O
experiments	O
.	O

Unsupervised	O
multiple	O
-	O
choice	O
question	O
generation	O
for	O
out	O
-	O
of	O
-	O
domain	O
Q&A	O
fine	O
-	O
tuning	O

Pre	O
-	O
trained	O
models	O
have	O
shown	O
very	O
good	O
performances	O
on	O
a	O
number	O
of	O
question	O
answering	O
benchmarks	O
especially	O
when	O
fine	O
-	O
tuned	O
on	O
multiple	O
question	O
answering	O
datasets	O
at	O
once	O
.	O
In	O
this	O
work	O
,	O
we	O
propose	O
an	O
approach	O
for	O
generating	O
a	O
fine	O
-	O
tuning	O
dataset	O
thanks	O
to	O
a	O
rule	O
-	O
based	O
algorithm	O
that	O
generates	O
questions	O
and	O
answers	O
from	O
unannotated	O
sentences	O
.	O
We	O
show	O
that	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
model	O
UnifiedQA	O
can	O
greatly	O
benefit	O
from	O
such	O
a	O
system	O
on	O
a	O
multiple	O
-	O
choice	O
benchmark	O
about	O
physics	O
,	O
biology	O
and	O
chemistry	O
it	O
has	O
never	O
been	O
trained	O
on	O
.	O
We	O
further	O
show	O
that	O
improved	O
performances	O
may	O
be	O
obtained	O
by	O
selecting	O
the	O
most	O
challenging	O
distractors	O
(	O
wrong	O
answers	O
)	O
,	O
with	O
a	O
dedicated	O
ranker	O
based	O
on	O
a	O
pretrained	O
RoBERTa	O
model	O
.	O

Introduction	O

In	O
the	O
past	O
years	O
,	O
deep	O
learning	O
models	O
have	O
greatly	O
improved	O
their	O
performances	O
on	O
a	O
large	O
range	O
of	O
question	O
answering	O
tasks	O
,	O
especially	O
using	O
pretrained	O
models	O
such	O
as	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
RoBERTa	O
(	O
Liu	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
T5	O
(	O
Raffel	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
More	O
recently	O
,	O
these	O
models	O
have	O
shown	O
even	O
better	O
performances	O
when	O
fine	O
-	O
tuned	O
on	O
multiple	O
question	O
answering	O
datasets	O
at	O
once	O
.	O
Such	O
a	O
model	O
is	O
UnifiedQA	O
(	O
Khashabi	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
which	O
,	O
starting	O
from	O
a	O
T5	O
model	O
,	O
is	O
trained	O
on	O
a	O
large	O
number	O
of	O
question	O
answering	O
datasets	O
including	O
multiple	O
choices	O
,	O
yes	O
/	O
no	O
,	O
extractive	O
and	O
abstractive	O
question	O
answering	O
.	O
UnifiedQA	O
is	O
,	O
at	O
the	O
time	O
of	O
writing	O
,	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
on	O
a	O
large	O
number	O
of	O
question	O
answering	O
datasets	O
including	O
multiple	O
-	O
choice	O
datasets	O
like	O
OpenBookQA	O
(	O
Mihaylov	O
et	O
al	O
.	O
,	O
2018	O
)	O
or	O
ARC	O
.	O
However	O
,	O
even	O
if	O
Uni	O
-	O
fiedQA	O
achieves	O
good	O
results	O
on	O
previously	O
unseen	O
datasets	O
,	O
it	O
often	O
fails	O
to	O
achieve	O
optimal	O
performances	O
on	O
these	O
datasets	O
until	O
it	O
is	O
further	O
finetuned	O
on	O
dedicated	O
human	O
annotated	O
data	O
.	O
This	O
tendency	O
is	O
increased	O
when	O
the	O
target	O
dataset	O
deals	O
with	O
questions	O
about	O
a	O
very	O
specific	O
domain	O
.	O

One	O
solution	O
to	O
this	O
problem	O
would	O
be	O
to	O
finetune	O
or	O
retrain	O
these	O
models	O
with	O
additionnal	O
human	O
annotated	O
data	O
.	O
However	O
,	O
this	O
is	O
expensive	O
both	O
in	O
time	O
and	O
resources	O
.	O
Instead	O
,	O
a	O
lot	O
of	O
work	O
has	O
been	O
done	O
lately	O
on	O
automatically	O
generating	O
training	O
data	O
for	O
fine	O
-	O
tuning	O
or	O
even	O
training	O
completely	O
unsupervised	O
models	O
for	O
question	O
answering	O
.	O
One	O
commonly	O
used	O
dataset	O
for	O
unsupervised	O
question	O
answering	O
is	O
the	O
extractive	O
dataset	O
SQUAD	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
proposed	O
a	O
question	O
generation	O
method	O
for	O
SQUAD	O
using	O
an	O
unsupervised	O
neural	O
based	O
translation	O
method	O
.	O
Fabbri	O
et	O
al	O
.	O
(	O
2020	O
)	O
and	O
further	O
gave	O
improved	O
unsupervised	O
performances	O
on	O
SQUAD	O
and	O
showed	O
that	O
simple	O
rulebased	O
question	O
generation	O
could	O
be	O
as	O
effective	O
as	O
the	O
previously	O
mentioned	O
neural	O
method	O
.	O
These	O
approches	O
are	O
rarely	O
applied	O
to	O
multiple	O
-	O
choice	O
questions	O
answering	O
in	O
part	O
due	O
to	O
the	O
difficulty	O
of	O
selecting	O
distractors	O
.	O
A	O
few	O
research	O
papers	O
however	O
proposed	O
distractor	O
selection	O
methods	O
for	O
multiple	O
-	O
choice	O
questions	O
using	O
either	O
supervised	O
approaches	O
(	O
Sakaguchi	O
et	O
al	O
.	O
,	O
2013;Liang	O
et	O
al	O
.	O
,	O
2018	O
)	O
or	O
general	O
purpose	O
knowledge	O
bases	O
(	O
Ren	O
and	O
Q.	O
Zhu	O
,	O
2021	O
)	O
.	O

In	O
this	O
paper	O
,	O
we	O
propose	O
an	O
unsupervised	O
process	O
to	O
generate	O
questions	O
,	O
answers	O
and	O
associated	O
distractors	O
in	O
order	O
to	O
fine	O
-	O
tune	O
and	O
improve	O
the	O
performance	O
of	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
model	O
UnifiedQA	O
on	O
unseen	O
domains	O
.	O
This	O
method	O
,	O
being	O
unsupervised	O
,	O
needs	O
no	O
additional	O
annotated	O
domain	O
specific	O
data	O
requiring	O
only	O
a	O
set	O
of	O
unannotated	O
sentences	O
of	O
the	O
domain	O
of	O
interest	O
from	O
which	O
the	O
questions	O
are	O
created	O
.	O
Contrarily	O
to	O
most	O
of	O
the	O
aforementioned	O
works	O
,	O
our	O
aim	O
is	O
not	O
to	O
train	O
a	O
new	O
completely	O
unsupervised	O
model	O
but	O
rather	O
to	O
incorporate	O
new	O
information	O
into	O
an	O
existing	O
stateof	O
-	O
the	O
-	O
art	O
model	O
and	O
thus	O
to	O
take	O
advantage	O
of	O
the	O
question	O
-	O
answering	O
knowledge	O
already	O
learned	O
.	O

We	O
conduct	O
our	O
experiments	O
on	O
the	O
SciQ	O
dataset	O
(	O
Welbl	O
et	O
al	O
.	O
,	O
2017	O
choice	O
questions	O
(	O
4	O
choices	O
)	O
featuring	O
subjects	O
centered	O
around	O
physics	O
,	O
biology	O
and	O
chemistry	O
.	O
An	O
example	O
of	O
question	O
can	O
be	O
found	O
in	O
Figure	O
1	O
.	O
We	O
focus	O
on	O
the	O
SciQ	O
dataset	O
because	O
it	O
has	O
not	O
yet	O
been	O
used	O
for	O
training	O
UnifiedQA	O
and	O
it	O
requires	O
precise	O
scientific	O
knowledge	O
.	O
Furthermore	O
,	O
our	O
experiments	O
reveal	O
that	O
the	O
direct	O
application	O
of	O
UnifiedQA	O
on	O
the	O
SciQ	O
benchmark	O
leads	O
to	O
a	O
much	O
lower	O
performance	O
than	O
when	O
fine	O
-	O
tuning	O
it	O
on	O
the	O
SciQ	O
training	O
set	O
(	O
see	O
Section	O
4	O
)	O
.	O
Our	O
objective	O
in	O
this	O
work	O
is	O
to	O
solve	O
this	O
gap	O
between	O
UnifiedQA	O
and	O
UnifiedQA	O
fine	O
-	O
tuned	O
on	O
supervised	O
data	O
with	O
the	O
unsupervised	O
question	O
generation	O
approach	O
described	O
in	O
Section	O
2	O
.	O
We	O
additionally	O
test	O
our	O
method	O
on	O
two	O
commonly	O
used	O
multiple	O
choice	O
question	O
answering	O
datasets	O
:	O
Common	O
-	O
senseQA	O
(	O
Talmor	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
QASC	O
.	O
These	O
datasets	O
contain	O
questions	O
with	O
similar	O
domains	O
to	O
SciQ	O
even	O
though	O
the	O
questions	O
are	O
slightly	O
less	O
specific	O
.	O
Furthermore	O
,	O
neither	O
of	O
them	O
has	O
been	O
used	O
during	O
the	O
initial	O
training	O
of	O
UnifiedQA	O
.	O

Question	O
Generation	O
Method	O

We	O
propose	O
a	O
method	O
for	O
generating	O
multiplechoice	O
questions	O
in	O
order	O
to	O
fine	O
-	O
tune	O
and	O
improve	O
UnifiedQA	O
.	O
This	O
process	O
is	O
based	O
on	O
3	O
steps	O
.	O
First	O
,	O
a	O
set	O
of	O
sentences	O
is	O
being	O
selected	O
(	O
Section	O
2.1	O
)	O
from	O
which	O
a	O
generic	O
question	O
generation	O
system	O
is	O
applied	O
(	O
Section	O
2.2	O
)	O
.	O
Then	O
a	O
number	O
of	O
distractors	O
are	O
added	O
to	O
each	O
question	O
(	O
Section	O
2.3	O
)	O
.	O

Sentence	O
Selection	O

Our	O
question	O
generation	O
method	O
uses	O
a	O
set	O
of	O
unannotated	O
sentences	O
from	O
which	O
the	O
questions	O
will	O
be	O
generated	O
.	O
We	O
compare	O
three	O
selection	O
methods	O
.	O
First	O
,	O
we	O
consider	O
a	O
scenario	O
where	O
the	O
application	O
developer	O
does	O
not	O
manually	O
collect	O
any	O
sentence	O
,	O
but	O
simply	O
gives	O
the	O
name	O
(	O
or	O
topic	O
)	O
of	O
the	O
target	O
domain	O
.	O
In	O
our	O
case	O
,	O
the	O
topics	O
are	O
"	O
Physics	O
"	O
,	O
"	O
Biology	O
"	O
and	O
"	O
Chemistry	O
"	O
since	O
these	O
are	O
the	O
main	O
domains	O
in	O
SciQ.	O
A	O
simple	O
information	O
retrieval	O
strategy	O
is	O
then	O
applied	O
to	O
automatically	O
mine	O
sentences	O
from	O
Wikipedia	O
.	O
We	O
first	O
compute	O
a	O
list	O
of	O
Wikipedia	O
categories	O
by	O
recursively	O
visiting	O
all	O
subcategories	O
starting	O
from	O
the	O
target	O
topic	O
names	O
.	O
The	O
maximum	O
recursion	O
number	O
is	O
limited	O
to	O
4	O
.	O
We	O
then	O
extract	O
the	O
summary	O
(	O
head	O
paragraph	O
of	O
each	O
Wikipedia	O
article	O
)	O
for	O
each	O
of	O
the	O
articles	O
matching	O
the	O
previously	O
extracted	O
categories	O
and	O
subcategories	O
.	O
We	O
only	O
keep	O
articles	O
with	O
more	O
than	O
800	O
average	O
visitors	O
per	O
day	O
for	O
the	O
last	O
ten	O
days	O
(	O
on	O
April	O
27	O
,	O
2021	O
)	O
,	O
resulting	O
in	O
12	O
656	O
pages	O
.	O

The	O
two	O
other	O
selection	O
methods	O
extract	O
sentences	O
from	O
SciQ	O
itself	O
and	O
therefore	O
are	O
not	O
entirely	O
unsupervised	O
but	O
rather	O
simulate	O
a	O
situation	O
where	O
we	O
have	O
access	O
to	O
unannotated	O
texts	O
that	O
precisely	O
describe	O
the	O
domains	O
of	O
interest	O
such	O
as	O
a	O
school	O
book	O
for	O
example	O
.	O
The	O
SciQ	O
dataset	O
includes	O
a	O
support	O
paragraph	O
for	O
each	O
question	O
(	O
see	O
Figure	O
1	O
)	O
.	O
Pooled	O
together	O
,	O
these	O
support	O
paragraphs	O
provide	O
us	O
with	O
a	O
large	O
dataset	O
of	O
texts	O
about	O
the	O
domains	O
of	O
interest	O
.	O
We	O
gather	O
the	O
paragraphs	O
corresponding	O
to	O
all	O
questions	O
and	O
split	O
them	O
into	O
sentences	O
to	O
produce	O
a	O
large	O
set	O
of	O
sentences	O
that	O
are	O
no	O
longer	O
associated	O
with	O
any	O
particular	O
question	O
but	O
cover	O
all	O
the	O
topics	O
found	O
in	O
the	O
questions	O
.	O
We	O
compare	O
two	O
different	O
setups	O
.	O
In	O
the	O
first	O
one	O
,	O
we	O
include	O
all	O
the	O
sentences	O
extracted	O
from	O
the	O
train	O
,	O
validation	O
and	O
test	O
sets	O
thus	O
simulating	O
a	O
perfect	O
selection	O
of	O
sentences	O
that	O
cover	O
all	O
the	O
knowledge	O
expressed	O
in	O
the	O
questions	O
.	O
Still	O
,	O
we	O
only	O
use	O
the	O
support	O
paragraphs	O
and	O
not	O
the	O
annotated	O
questions	O
themselves	O
.	O
As	O
compared	O
to	O
the	O
classical	O
supervised	O
paradigm	O
,	O
this	O
setting	O
removes	O
all	O
annotation	O
costs	O
for	O
the	O
application	O
developer	O
,	O
but	O
it	O
still	O
requires	O
to	O
gather	O
sentences	O
that	O
are	O
deemed	O
useful	O
for	O
the	O
test	O
set	O
of	O
interest	O
.	O
We	O
then	O
compare	O
this	O
setup	O
with	O
another	O
one	O
,	O
where	O
only	O
the	O
sentences	O
from	O
the	O
train	O
set	O
are	O
included	O
.	O
This	O
scenario	O
arguably	O
meets	O
more	O
practical	O
needs	O
since	O
it	O
would	O
suffice	O
to	O
gather	O
sentences	O
close	O
to	O
the	O
domain	O
of	O
interest	O
.	O
The	O
number	O
of	O
sentences	O
for	O
each	O
dataset	O
is	O
presented	O
in	O
Table	O
1	O
.	O

Questions	O
Generation	O

The	O
generation	O
of	O
questions	O
from	O
a	O
sentence	O
relies	O
on	O
the	O
jsRealB	O
text	O
realizer	O
(	O
Lapalme	O
,	O
2021	O
)	O
which	O
generates	O
an	O
affirmative	O
sentence	O
from	O
a	O
constituent	O
structure	O
.	O
It	O
can	O
also	O
be	O
parameterized	O
to	O
generate	O
variations	O
of	O
the	O
original	O
sentence	O
such	O
as	O
its	O
negation	O
,	O
its	O
passive	O
form	O
and	O
different	O
types	O
of	O
questions	O
such	O
as	O
who	O
,	O
what	O
,	O
when	O
,	O
etc	O
.	O
The	O
constituency	O
structure	O
of	O
a	O
sentence	O
is	O
most	O
often	O
created	O
by	O
a	O
user	O
or	O
by	O
a	O
program	O
from	O
data	O
.	O
In	O
this	O
work	O
,	O
it	O
is	O
instead	O
built	O
from	O
a	O
Universal	O
Dependency	O
(	O
UD	O
)	O
structure	O
using	O
a	O
technique	O
developed	O
for	O
SR'19	O
(	O
Lapalme	O
,	O
2019	O
)	O
.	O
The	O
UD	O
structure	O
of	O
a	O
sentence	O
is	O
the	O
result	O
of	O
a	O
dependency	O
parse	O
with	O
Stanza	O
(	O
Qi	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
We	O
thus	O
have	O
a	O
pipeline	O
composed	O
of	O
a	O
neural	O
dependency	O
parser	O
,	O
followed	O
by	O
a	O
program	O
to	O
create	O
a	O
constituency	O
structure	O
used	O
as	O
input	O
for	O
a	O
text	O
realizer	O
,	O
both	O
in	O
JavaScript	O
.	O
Used	O
without	O
modification	O
,	O
this	O
would	O
create	O
a	O
complex	O
echo	O
program	O
for	O
the	O
original	O
affirmative	O
sentence	O
,	O
but	O
by	O
changing	O
parameters	O
,	O
its	O
output	O
can	O
vary	O
.	O

In	O
order	O
to	O
create	O
questions	O
from	O
a	O
single	O
constituency	O
structure	O
,	O
jsRealB	O
uses	O
the	O
classical	O
grammar	O
transformations	O
:	O
for	O
a	O
who	O
question	O
,	O
it	O
removes	O
the	O
subject	O
(	O
i.e.	O
the	O
first	O
noun	O
phrase	O
before	O
the	O
verb	O
phrase	O
)	O
,	O
for	O
a	O
what	O
question	O
,	O
it	O
removes	O
the	O
subject	O
or	O
the	O
direct	O
object	O
(	O
i.e.	O
the	O
first	O
noun	O
phrase	O
within	O
the	O
verb	O
phrase	O
)	O
;	O
for	O
other	O
types	O
of	O
questions	O
(	O
when	O
,	O
where	O
)	O
it	O
removes	O
the	O
first	O
prepositional	O
phrase	O
within	O
the	O
verb	O
phrase	O
.	O
Depending	O
on	O
the	O
preposition	O
,	O
the	O
question	O
will	O
be	O
a	O
when	O
or	O
a	O
where	O
.	O
Note	O
that	O
the	O
removed	O
part	O
becomes	O
the	O
answer	O
to	O
the	O
question	O
.	O

In	O
order	O
to	O
determine	O
which	O
questions	O
are	O
appropriate	O
for	O
a	O
given	O
sentence	O
,	O
we	O
examine	O
the	O
dependency	O
structure	O
of	O
the	O
original	O
sentence	O
and	O
check	O
if	O
it	O
contains	O
the	O
required	O
part	O
to	O
be	O
removed	O
before	O
parameterizing	O
the	O
realization	O
.	O
The	O
generated	O
questions	O
are	O
then	O
filtered	O
to	O
remove	O
any	O
question	O
for	O
which	O
the	O
answer	O
is	O
composed	O
of	O
a	O
single	O
stopword	O
.	O
Table	O
1	O
shows	O
the	O
number	O
of	O
questions	O
generated	O
for	O
each	O
dataset	O
.	O
An	O
example	O
of	O
a	O
synthetic	O
question	O
is	O
shown	O
in	O
Figure	O
3	O
.	O

Distractors	O
Selection	O

Since	O
SciQ	O
is	O
a	O
multiple	O
-	O
choice	O
dataset	O
,	O
we	O
must	O
add	O
distractors	O
to	O
each	O
question	O
we	O
generate	O
,	O
to	O
match	O
the	O
format	O
of	O
SciQ.	O
A	O
simple	O
solution	O
to	O
this	O
problem	O
is	O
to	O
select	O
random	O
distractors	O
among	O
answers	O
to	O
other	O
similar	O
questions	O
generated	O
from	O
the	O
dataset	O
of	O
sentences	O
we	O
gathered	O
.	O
Obviously	O
,	O
selecting	O
random	O
distractors	O
may	O
lead	O
to	O
a	O
fine	O
-	O
tuning	O
dataset	O
that	O
is	O
too	O
easy	O
to	O
solve	O
.	O
Therefore	O
,	O
we	O
propose	O
another	O
strategy	O
that	O
selects	O
hard	O
distractors	O
for	O
each	O
question	O
.	O
To	O
do	O
so	O
,	O
starting	O
from	O
our	O
synthetic	O
dataset	O
with	O
random	O
distractors	O
,	O
we	O
finetune	O
RoBERTa	O
(	O
Liu	O
et	O
al	O
.	O
,	O
2019	O
)	O
using	O
the	O
standard	O
method	O
of	O
training	O
for	O
multiple	O
choices	O
question	O
answering	O
.	O
Each	O
pair	O
question	O
/	O
choice	O
is	O
fed	O
to	O
RoBERTa	O
and	O
the	O
embedding	O
corresponding	O
to	O
the	O
first	O
token	O
(	O
"	O
[	O
CLS	O
]	O
"	O
)	O
is	O
given	O
to	O
a	O
linear	O
layer	O
to	O
produce	O
a	O
single	O
scalar	O
score	O
for	O
each	O
choice	O
.	O
The	O
scores	O
corresponding	O
to	O
every	O
choice	O
for	O
a	O
given	O
question	O
are	O
then	O
compared	O
to	O
each	O
other	O
by	O
a	O
softmax	O
and	O
a	O
cross	O
-	O
entropy	O
loss	O
.	O
With	O
this	O
method	O
,	O
RoBERTa	O
is	O
trained	O
to	O
score	O
a	O
possible	O
answer	O
for	O
a	O
given	O
question	O
,	O
based	O
on	O
whether	O
or	O
not	O
it	O
is	O
a	O
credible	O
answer	O
to	O
that	O
question	O
.	O
For	O
each	O
question	O
,	O
we	O
then	O
randomly	O
select	O
a	O
number	O
of	O
candidate	O
distractors	O
from	O
the	O
answers	O
to	O
other	O
questions	O
and	O
we	O
use	O
our	O
trained	O
RoBERTa	O
to	O
score	O
each	O
of	O
these	O
candidates	O
.	O
The	O
3	O
candidates	O
with	O
the	O
highest	O
scores	O
(	O
and	O
thus	O
the	O
most	O
credible	O
answers	O
)	O
are	O
selected	O
.	O
The	O
idea	O
is	O
that	O
during	O
this	O
first	O
training	O
,	O
RoBERTa	O
will	O
learn	O
a	O
large	O
amount	O
of	O
simplistic	O
logic	O
.	O
For	O
example	O
,	O
because	O
of	O
the	O
initial	O
random	O
selection	O
of	O
distractors	O
,	O
it	O
is	O
highly	O
unlikely	O
that	O
even	O
one	O
of	O
the	O
distractors	O
will	O
be	O
close	O
enough	O
to	O
the	O
question	O
's	O
semantic	O
field	O
.	O
Furthermore	O
,	O
a	O
lot	O
distractors	O
have	O
an	O
incorrect	O
grammar	O
(	O
eg	O
:	O
a	O
distractor	O
might	O
be	O
plural	O
when	O
the	O
question	O
expects	O
a	O
singular	O
)	O
.	O
Therefore	O
,	O
in	O
this	O
initial	O
training	O
,	O
RoBERTa	O
might	O
learn	O
to	O
isolate	O
the	O
answer	O
with	O
a	O
corresponding	O
semantic	O
field	O
or	O
the	O
one	O
with	O
correct	O
grammar	O
.	O
The	O
re	O
-	O
selection	O
then	O
minimizes	O
the	O
amount	O
of	O
trivial	O
distractors	O
and	O
models	O
trained	O
on	O
this	O
new	O
refined	O
dataset	O
will	O
have	O
to	O
focus	O
on	O
deeper	O
and	O
more	O
meaningful	O
relations	O
between	O
the	O
questions	O
and	O
the	O
answers	O
.	O
The	O
process	O
is	O
better	O
shown	O
in	O
Figure	O
4	O
,	O
and	O
an	O
example	O
of	O
refined	O
distractors	O
can	O
be	O
found	O
in	O
Figure	O
3	O
.	O

The	O
number	O
of	O
scored	O
candidate	O
distractors	O
is	O
an	O
hyper	O
-	O
parameter	O
.	O
A	O
small	O
number	O
of	O
candidates	O
may	O
result	O
in	O
a	O
situation	O
where	O
none	O
of	O
the	O
candidates	O
are	O
credible	O
enough	O
,	O
while	O
a	O
large	O
number	O
requires	O
more	O
computation	O
time	O
,	O
since	O
the	O
score	O
of	O
each	O
candidate	O
for	O
every	O
question	O
needs	O
to	O
be	O
computed	O
,	O
and	O
has	O
a	O
higher	O
risk	O
of	O
proposing	O
multiple	O
valid	O
answers	O
.	O
In	O
our	O
experiments	O
,	O
we	O
use	O
a	O
number	O
of	O
64	O
candidates	O
in	O
order	O
to	O
limit	O
computation	O
time	O
.	O

Training	O
and	O
Implementation	O
Details	O

To	O
refine	O
distractors	O
,	O
we	O
use	O
the	O
"	O
Large	O
"	O
version	O
of	O
RoBERTa	O
and	O
all	O
models	O
are	O
trained	O
for	O
4	O
epochs	O
and	O
a	O
learning	O
rate	O
of	O
1	O
×	O
10	O
−5	O
.	O
These	O
hyperparameters	O
are	O
chosen	O
based	O
on	O
previous	O
experiments	O
with	O
RoBERTa	O
on	O
other	O
multiple	O
-	O
choice	O
datasets	O
.	O
The	O
final	O
UnifiedQA	O
fine	O
-	O
tuning	O
is	O
done	O
using	O
the	O
same	O
multiple	O
choices	O
question	O
answering	O
setup	O
as	O
the	O
one	O
used	O
in	O
the	O
original	O
UnifiedQA	O
paper	O
(	O
Khashabi	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
We	O
use	O
the	O
"	O
Large	O
"	O
version	O
of	O
UnifiedQA	O
and	O
all	O
the	O
models	O
are	O
trained	O
for	O
4	O
epochs	O
using	O
Adafactor	O
and	O
a	O
learning	O
rate	O
of	O
1	O
×	O
10	O
−5	O
.	O
The	O
learning	O
rate	O
is	O
loosely	O
tuned	O
to	O
get	O
the	O
best	O
performance	O
on	O
the	O
validation	O
set	O
during	O
the	O
supervised	O
training	O
of	O
UnifiedQA	O
.	O
We	O
use	O
the	O
Hugging	O
Face	O
pytorch	O
-	O
transformers	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2020	O
)	O
library	O
for	O
model	O
implementation	O
.	O
Experiments	O
presented	O
in	O
this	O
paper	O
were	O
carried	O
out	O
using	O
the	O
Grid'5000	O
testbed	O
(	O
Balouek	O
et	O
al	O
.	O
,	O
2013	O
)	O
,	O
supported	O
by	O
a	O
scientific	O
interest	O
group	O
hosted	O
by	O
Inria	O
and	O
including	O
CNRS	O
,	O
RENATER	O
and	O
several	O
Universities	O
as	O
well	O
as	O
other	O
organizations	O
(	O
see	O
https://www.grid5000.fr	O
)	O
.	O

Results	O

Accuracy	O
results	O
in	O
Table	O
2	O
have	O
a	O
95	O
%	O
Wald	O
confidence	O
interval	O
of	O
±2.8	O
%	O
.	O
The	O
first	O
row	O
of	O
Table	O
2	O
presents	O
the	O
accuracy	O
results	O
of	O
a	O
vanilla	O
UnifiedQA	O
large	O
model	O
on	O
SciQ.	O
The	O
second	O
line	O
shows	O
the	O
accuracy	O
when	O
UnifiedQA	O
is	O
fine	O
-	O
tuned	O
over	O
the	O
full	O
training	O
corpus	O
.	O
Our	O
objective	O
is	O
thus	O
to	O
get	O
as	O
close	O
as	O
possible	O
to	O
this	O
accuracy	O
score	O
using	O
only	O
un	O
-	O
supervised	O
methods	O
.	O
The	O
results	O
using	O
Wikipedia	O
are	O
the	O
only	O
ones	O
that	O
are	O
unsupervised	O
and	O
therefore	O
are	O
the	O
ones	O
directly	O
comparable	O
to	O
UnifiedQA	O
with	O
no	O
fine	O
-	O
tuning	O
or	O
other	O
unsupervised	O
methods	O
.	O
Table	O
2	O
:	O
Accuracy	O
on	O
SciQ	O
by	O
UnifiedQA	O
fine	O
-	O
tuned	O
on	O
our	O
synthetic	O
datasets	O
.	O
"	O
SciQ	O
data	O
"	O
refers	O
to	O
the	O
questions	O
generated	O
using	O
the	O
support	O
paragraphs	O
in	O
SciQ	O
while	O
"	O
Wikipedia	O
data	O
"	O
refers	O
to	O
questions	O
generated	O
using	O
sentences	O
harvested	O
from	O
Wikipedia	O
.	O
All	O
scores	O
are	O
averaged	O
over	O
3	O
independent	O
runs	O
(	O
including	O
the	O
complete	O
question	O
generation	O
process	O
and	O
the	O
final	O
Uni	O
-	O
fiedQA	O
fine	O
-	O
tuning	O
)	O
.	O

Fine	O
-	O
tuning	O
UnifiedQA	O
on	O
synthetic	O
questions	O
with	O
random	O
distractors	O
improves	O
the	O
results	O
as	O
compared	O
to	O
the	O
baseline	O
and	O
,	O
as	O
expected	O
,	O
the	O
closer	O
the	O
unlabeled	O
sentences	O
are	O
to	O
the	O
topics	O
of	O
the	O
questions	O
,	O
the	O
better	O
is	O
the	O
accuracy	O
.	O
Hence	O
,	O
generating	O
questions	O
from	O
only	O
the	O
train	O
set	O
of	O
SciQ	O
gives	O
performances	O
that	O
are	O
comparable	O
but	O
slightly	O
lower	O
to	O
the	O
ones	O
obtained	O
from	O
the	O
combined	O
train	O
,	O
dev	O
and	O
test	O
set	O
of	O
SciQ.	O
Finally	O
,	O
questions	O
selected	O
from	O
Wikipedia	O
also	O
improve	O
the	O
results	O
,	O
despite	O
being	O
loosely	O
related	O
to	O
the	O
target	O
test	O
corpus	O
.	O
Our	O
distractor	O
selection	O
method	O
further	O
boosts	O
the	O
accuracy	O
results	O
in	O
all	O
setups	O
.	O
This	O
suggests	O
that	O
a	O
careful	O
selection	O
of	O
distractors	O
is	O
important	O
,	O
and	O
that	O
the	O
hard	O
selection	O
criterion	O
used	O
here	O
seems	O
adequate	O
in	O
our	O
context	O
.	O

The	O
results	O
for	O
CommonsenseQA	O
and	O
QASC	O
using	O
the	O
same	O
selection	O
of	O
sentences	O
from	O
Wikipedia	O
are	O
reported	O
in	O
table	O
3	O
.	O
Overall	O
,	O
we	O
obtain	O
similar	O
results	O
to	O
SciQ	O
with	O
a	O
large	O
improvement	O
of	O
performances	O
when	O
generating	O
questions	O
and	O
a	O
further	O
boost	O
with	O
refined	O
distractors	O
.	O
However	O
compared	O
to	O
SciQ	O
,	O
the	O
improvement	O
brought	O
by	O
the	O
distractor	O
refining	O
process	O
is	O
less	O
significant	O
.	O
This	O
could	O
be	O
partly	O
explained	O
by	O
the	O
fact	O
that	O
the	O
distractors	O
in	O
the	O
original	O
QASC	O
and	O
CommonsenseQA	O
datasets	O
are	O
overall	O
easier	O
and	O
therefore	O
it	O
is	O
less	O
advantageous	O
for	O
a	O
model	O
to	O
be	O
trained	O
on	O
harder	O
questions	O
.	O

Conclusion	O

In	O
this	O
work	O
,	O
we	O
proposed	O
a	O
multiple	O
-	O
choice	O
question	O
generation	O
method	O
that	O
can	O
be	O
used	O
to	O
fine	O
-	O
tune	O
the	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
UnifiedQA	O
model	O
and	O
improve	O
its	O
performance	O
on	O
an	O
unseen	O
and	O
out	O
of	O
domain	O
dataset	O
.	O
Our	O
contributions	O
are	O
:	O

•	O
We	O
have	O
shown	O
that	O
simple	O
unsupervised	O
methods	O
could	O
be	O
used	O
to	O
finetune	O
existing	O
multipurpose	O
question	O
answering	O
models	O
(	O
in	O
our	O
case	O
UnifiedQA	O
)	O
to	O
new	O
datasets	O
or	O
domains	O
.	O

•	O
We	O
propose	O
a	O
novel	O
distractor	O
refining	O
method	O
able	O
to	O
select	O
harder	O
distractors	O
for	O
a	O
given	O
generated	O
question	O
and	O
show	O
its	O
superiority	O
compared	O
to	O
a	O
random	O
selection	O
.	O

Future	O
work	O
includes	O
comparing	O
our	O
method	O
to	O
other	O
question	O
generation	O
methods	O
(	O
including	O
supervised	O
methods	O
:	O
,	O
Puri	O
et	O
al	O
.	O
(	O
2020	O
)	O
)	O
in	O
order	O
to	O
assess	O
the	O
effect	O
of	O
both	O
the	O
generation	O
method	O
and	O
the	O
questions	O
quality	O
on	O
the	O
final	O
performances	O
of	O
our	O
models	O
.	O
Also	O
,	O
we	O
will	O
further	O
compare	O
different	O
variations	O
of	O
our	O
question	O
generation	O
and	O
distractor	O
refining	O
methods	O
in	O
order	O
to	O
more	O
thoroughly	O
understand	O
the	O
effect	O
of	O
hyper	O
-	O
parameters	O
such	O
as	O
the	O
number	O
of	O
candidate	O
distractors	O
.	O

Global	O
Entity	O
Disambiguation	O
with	O
BERT	O

We	O
propose	O
a	O
global	O
entity	O
disambiguation	O
(	O
ED	O
)	O
model	O
based	O
on	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
To	O
capture	O
global	O
contextual	O
information	O
for	O
ED	O
,	O
our	O
model	O
treats	O
not	O
only	O
words	O
but	O
also	O
entities	O
as	O
input	O
tokens	O
,	O
and	O
solves	O
the	O
task	O
by	O
sequentially	O
resolving	O
mentions	O
to	O
their	O
referent	O
entities	O
and	O
using	O
resolved	O
entities	O
as	O
inputs	O
at	O
each	O
step	O
.	O
We	O
train	O
the	O
model	O
using	O
a	O
large	O
entity	O
-	O
annotated	O
corpus	O
obtained	O
from	O
Wikipedia	O
.	O
We	O
achieve	O
new	O
state	O
-	O
of	O
-	O
the	O
-	O
art	O
results	O
on	O
five	O
standard	O
ED	O
datasets	O
:	O
AIDA	O
-	O
CoNLL	O
,	O
MSNBC	O
,	O
AQUAINT	O
,	O
ACE2004	O
,	O
and	O
WNED	O
-	O
WIKI	O
.	O
The	O
source	O
code	O
and	O
model	O
checkpoint	O
are	O
available	O
at	O
https	O
:	O
//github.com	O
/	O
studio	O
-	O
ousia	O
/	O
luke	O
.	O

Introduction	O

Entity	O
disambiguation	O
(	O
ED	O
)	O
refers	O
to	O
the	O
task	O
of	O
assigning	O
mentions	O
in	O
a	O
document	O
to	O
corresponding	O
entities	O
in	O
a	O
knowledge	O
base	O
(	O
KB	O
)	O
.	O
This	O
task	O
is	O
challenging	O
because	O
of	O
the	O
ambiguity	O
between	O
mentions	O
(	O
e.g.	O
,	O
"	O
World	O
Cup	O
"	O
)	O
and	O
the	O
entities	O
they	O
refer	O
to	O
(	O
e.g.	O
,	O
FIFA	O
World	O
Cup	O
or	O
Rugby	O
World	O
Cup	O
)	O
.	O
ED	O
models	O
typically	O
rely	O
on	O
local	O
contextual	O
information	O
based	O
on	O
words	O
that	O
co	O
-	O
occur	O
with	O
the	O
mention	O
and	O
global	O
contextual	O
information	O
based	O
on	O
the	O
entity	O
-	O
based	O
coherence	O
of	O
the	O
disambiguation	O
decisions	O
.	O
A	O
key	O
to	O
improve	O
the	O
performance	O
of	O
ED	O
is	O
to	O
effectively	O
combine	O
both	O
local	O
and	O
global	O
contextual	O
information	O
(	O
Ganea	O
and	O
Hofmann	O
,	O
2017;Le	O
and	O
Titov	O
,	O
2018	O
)	O
.	O

In	O
this	O
study	O
,	O
we	O
propose	O
a	O
global	O
ED	O
model	O
based	O
on	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Our	O
model	O
treats	O
words	O
and	O
entities	O
in	O
the	O
document	O
as	O
input	O
tokens	O
,	O
and	O
is	O
trained	O
by	O
predicting	O
randomly	O
masked	O
entities	O
in	O
a	O
large	O
entity	O
-	O
annotated	O
corpus	O
obtained	O
from	O
Wikipedia	O
.	O
This	O
training	O
enables	O
the	O
model	O
to	O
learn	O
how	O
to	O
disambiguate	O
masked	O
entities	O
based	O
on	O
words	O
and	O
non	O
-	O
masked	O
entities	O
.	O
At	O
the	O
inference	O
time	O
,	O
our	O
model	O
disambiguates	O
*	O
Work	O
done	O
at	O
RIKEN	O
.	O

mentions	O
sequentially	O
using	O
words	O
and	O
already	O
resolved	O
entities	O
(	O
see	O
Figure	O
1	O
)	O
.	O
This	O
sequential	O
inference	O
effectively	O
accumulates	O
the	O
global	O
contextual	O
information	O
and	O
enhances	O
the	O
coherence	O
of	O
disambiguation	O
decisions	O
.	O
We	O
conducted	O
extensive	O
experiments	O
using	O
six	O
standard	O
ED	O
datasets	O
,	O
i.e.	O
,	O
AIDA	O
-	O
CoNLL	O
,	O
MSNBC	O
,	O
AQUAINT	O
,	O
ACE2004	O
,	O
WNED	O
-	O
WIKI	O
,	O
and	O
WNED	O
-	O
CWEB	O
.	O
As	O
a	O
result	O
,	O
the	O
global	O
contextual	O
information	O
consistently	O
improved	O
the	O
performance	O
.	O
Furthermore	O
,	O
we	O
achieved	O
new	O
state	O
of	O
the	O
art	O
on	O
all	O
datasets	O
except	O
for	O
WNED	O
-	O
CWEB	O
.	O
The	O
source	O
code	O
and	O
model	O
checkpoint	O
are	O
available	O
at	O
https://github.com/	O
studio	O
-	O
ousia	O
/	O
luke	O
.	O

Related	O
Work	O

Transformer	O
-	O
based	O
ED	O
.	O
Several	O
recent	O
studies	O
have	O
proposed	O
ED	O
models	O
based	O
on	O
Transformer	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
trained	O
with	O
a	O
large	O
entity	O
-	O
annotated	O
corpus	O
obtained	O
from	O
Wikipedia	O
(	O
Broscheit	O
,	O
2019;Ling	O
et	O
al	O
.	O
,	O
2020;Cao	O
et	O
al	O
.	O
,	O
2021;Barba	O
et	O
al	O
.	O
,	O
2022	O
)	O
.	O
Broscheit	O
(	O
2019	O
)	O
trained	O
an	O
ED	O
model	O
based	O
on	O
BERT	O
by	O
classifying	O
each	O
word	O
in	O
the	O
document	O
to	O
the	O
corresponding	O
entity	O
.	O
Similarly	O
,	O
addressed	O
ED	O
using	O
BERT	O
by	O
classifying	O
mention	O
spans	O
to	O
the	O
corresponding	O
entities	O
.	O
Ling	O

Words	O
Entities	O

Input	O

Position	O
emb	O
.	O
(	O
Lewis	O
et	O
al	O
.	O
,	O
2020	O
)	O
to	O
generate	O
referent	O
entity	O
titles	O
of	O
target	O
mentions	O
in	O
an	O
autoregressive	O
manner	O
.	O
Barba	O
et	O
al	O
.	O
(	O
2022	O
)	O
formulated	O
ED	O
as	O
a	O
text	O
extraction	O
problem	O
;	O
they	O
fed	O
the	O
document	O
and	O
candidate	O
entity	O
titles	O
to	O
BART	O
and	O
Longformer	O
(	O
Beltagy	O
et	O
al	O
.	O
,	O
2020	O
)	O
and	O
disambiguated	O
a	O
mention	O
in	O
the	O
document	O
by	O
extracting	O
the	O
referent	O
entity	O
title	O
of	O
the	O
mention	O
.	O
However	O
,	O
unlike	O
our	O
model	O
,	O
these	O
models	O
addressed	O
the	O
task	O
based	O
only	O
on	O
local	O
contextual	O
information	O
.	O

Treating	O
entities	O
as	O
inputs	O
of	O
Transformer	O
.	O
Recent	O
studies	O
Yamada	O
et	O
al	O
.	O
,	O
2020;Sun	O
et	O
al	O
.	O
,	O
2020	O
)	O
have	O
proposed	O
Transformerbased	O
models	O
that	O
treat	O
entities	O
as	O
input	O
tokens	O
to	O
enrich	O
their	O
expressiveness	O
using	O
additional	O
information	O
contained	O
in	O
the	O
entity	O
embeddings	O
.	O
However	O
,	O
these	O
models	O
were	O
designed	O
to	O
solve	O
general	O
NLP	O
tasks	O
and	O
not	O
tested	O
on	O
ED	O
.	O
We	O
treat	O
entities	O
as	O
input	O
tokens	O
to	O
capture	O
the	O
global	O
context	O
that	O
is	O
shown	O
to	O
be	O
highly	O
effective	O
for	O
ED	O
.	O

ED	O
as	O
sequential	O
decision	O
task	O
.	O
Past	O
studies	O
Fang	O
et	O
al	O
.	O
,	O
2019	O
)	O
have	O
solved	O
ED	O
by	O
casting	O
it	O
as	O
a	O
sequential	O
decision	O
task	O
to	O
capture	O
global	O
contextual	O
information	O
.	O
We	O
adopt	O
a	O
similar	O
method	O
with	O
an	O
enhanced	O
Transformer	O
architecture	O
,	O
a	O
training	O
task	O
,	O
and	O
an	O
inference	O
method	O
to	O
implement	O
the	O
global	O
ED	O
model	O
based	O
on	O
BERT	O
.	O

Model	O

Given	O
a	O
document	O
with	O
N	O
mentions	O
,	O
each	O
of	O
which	O
has	O
K	O
entity	O
candidates	O
,	O
our	O
model	O
solves	O
ED	O
by	O
selecting	O
a	O
correct	O
referent	O
entity	O
from	O
the	O
entity	O
candidates	O
for	O
each	O
mention	O
.	O

Model	O
Architecture	O

Our	O
model	O
is	O
based	O
on	O
BERT	O
and	O
takes	O
words	O
and	O
entities	O
(	O
Wikipedia	O
entities	O
or	O
the	O
[	O
MASK	O
]	O
entity	O
)	O
.	O

The	O
input	O
representation	O
of	O
a	O
word	O
or	O
an	O
entity	O
is	O
constructed	O
by	O
summing	O
the	O
token	O
,	O
token	O
type	O
,	O
and	O
position	O
embeddings	O
(	O
see	O
Figure	O
2	O
):	O

Token	O
embedding	O
is	O
the	O
embedding	O
of	O
the	O
corresponding	O
token	O
.	O
The	O
matrices	O
of	O
the	O
word	O
and	O
entity	O
token	O
embeddings	O
are	O
represented	O
as	O
A	O
∈	O
R	O
Vw×H	O
and	O
B	O
∈	O
R	O
Ve×H	O
,	O
respectively	O
,	O
where	O
H	O
is	O
the	O
size	O
of	O
the	O
hidden	O
states	O
of	O
BERT	O
,	O
and	O
V	O
w	O
and	O
V	O
e	O
are	O
the	O
number	O
of	O
items	O
in	O
the	O
word	O
vocabulary	O
and	O
that	O
of	O
the	O
entity	O
vocabulary	O
,	O
respectively	O
.	O

Token	O
type	O
embedding	O
represents	O
the	O
type	O
of	O
token	O
,	O
namely	O
word	O
(	O
C	O
word	O
)	O
or	O
entity	O
(	O
C	O
entity	O
)	O
.	O

Position	O
embedding	O
represents	O
the	O
position	O
of	O
the	O
token	O
in	O
a	O
word	O
sequence	O
.	O
A	O
word	O
and	O
an	O
entity	O
appearing	O
at	O
the	O
i	O
-	O
th	O
position	O
in	O
the	O
sequence	O
are	O
represented	O
as	O
D	O
i	O
and	O
E	O
i	O
,	O
respectively	O
.	O
If	O
an	O
entity	O
mention	O
contains	O
multiple	O
words	O
,	O
its	O
position	O
embedding	O
is	O
computed	O
by	O
averaging	O
the	O
embeddings	O
of	O
the	O
corresponding	O
positions	O
(	O
see	O
Figure	O
2	O
)	O
.	O
Following	O
Devlin	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
we	O
tokenize	O
the	O
document	O
text	O
using	O
the	O
BERT	O
's	O
wordpiece	O
tokenizer	O
,	O
and	O
insert	O
[	O
CLS	O
]	O
and	O
[	O
SEP	O
]	O
tokens	O
as	O
the	O
first	O
and	O
last	O
words	O
,	O
respectively	O
.	O

Training	O
Task	O

Similar	O
to	O
the	O
masked	O
language	O
model	O
(	O
MLM	O
)	O
objective	O
adopted	O
in	O
BERT	O
,	O
our	O
model	O
is	O
trained	O
by	O
predicting	O
randomly	O
masked	O
entities	O
.	O
Specifically	O
,	O
we	O
randomly	O
replace	O
some	O
percentage	O
of	O
the	O
entities	O
with	O
special	O
[	O
MASK	O
]	O
entity	O
tokens	O
and	O
then	O
trains	O
the	O
model	O
to	O
predict	O
masked	O
entities	O
.	O

We	O
adopt	O
a	O
model	O
equivalent	O
to	O
the	O
one	O
used	O
to	O
predict	O
words	O
in	O
MLM	O
.	O
Formally	O
,	O
we	O
predict	O
the	O
original	O
entity	O
corresponding	O
to	O
a	O
masked	O
entity	O
by	O
applying	O
softmax	O
over	O
all	O
entities	O
:	O

ED	O
Model	O

Local	O
ED	O
Model	O
.	O
Our	O
local	O
ED	O
model	O
takes	O
words	O
and	O
N	O
[	O
MASK	O
]	O
tokens	O
corresponding	O
to	O
the	O
mentions	O
in	O
the	O
document	O
.	O
The	O
model	O
then	O
computes	O
the	O
embedding	O
m	O
′	O
e	O
∈	O
R	O
H	O
for	O
each	O
[	O
MASK	O
]	O
token	O
using	O
Eq.(2	O
)	O
and	O
predicts	O
the	O
entity	O
using	O
softmax	O
over	O
the	O
K	O
entity	O
candidates	O
:	O

where	O
B	O
*	O
∈	O
R	O
K×H	O
and	O
b	O
*	O
o	O
∈	O
R	O
K	O
consist	O
of	O
the	O
entity	O
token	O
embeddings	O
and	O
the	O
bias	O
corresponding	O
to	O
the	O
entity	O
candidates	O
,	O
respectively	O
.	O
Note	O
that	O
B	O
*	O
and	O
b	O
*	O
o	O
are	O
the	O
subsets	O
of	O
B	O
and	O
b	O
o	O
,	O
respectively	O
.	O
Global	O
ED	O
Model	O
.	O
Our	O
global	O
ED	O
model	O
resolves	O
mentions	O
sequentially	O
for	O
N	O
steps	O
(	O
see	O
Algorithm	O
1	O
)	O
.	O
First	O
,	O
the	O
model	O
initializes	O
the	O
entity	O
of	O
each	O
mention	O
using	O
the	O
[	O
MASK	O
]	O
token	O
.	O
Then	O
,	O
for	O
each	O
step	O
,	O
it	O
predicts	O
an	O
entity	O
for	O
each	O
[	O
MASK	O
]	O
token	O
,	O
selects	O
the	O
prediction	O
with	O
the	O
highest	O
probability	O
produced	O
by	O
the	O
softmax	O
function	O
in	O
Eq.(3	O
)	O
,	O
and	O
resolves	O
the	O
corresponding	O
mention	O
by	O
assigning	O
the	O
predicted	O
entity	O
to	O
it	O
.	O
This	O
model	O
is	O
denoted	O
as	O
confidence	O
-	O
order	O
.	O
We	O
also	O
test	O
a	O
model	O
that	O
selects	O
mentions	O
according	O
to	O
their	O
order	O
of	O
appearance	O
in	O
the	O
document	O
and	O
denote	O
it	O
by	O
natural	O
-	O
order	O
.	O

Modeling	O
Details	O

Our	O
model	O
is	O
based	O
on	O
BERT	O
LARGE	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
The	O
parameters	O
shared	O
with	O
BERT	O
are	O
initialized	O
using	O
BERT	O
,	O
and	O
the	O
other	O
parameters	O
are	O
initialized	O
randomly	O
.	O
We	O
treat	O
the	O
hyperlinks	O
in	O
Wikipedia	O
as	O
entity	O
annotations	O
and	O
randomly	O
mask	O
30	O
%	O
of	O
all	O
entities	O
.	O
We	O
train	O
the	O
model	O
by	O
maximizing	O
the	O
log	O
likelihood	O
of	O
entity	O
predictions	O
.	O
Further	O
details	O
are	O
described	O
in	O
Appendix	O
A.	O

Experiments	O

Our	O
experimental	O
setup	O
follows	O
Le	O
and	O
Titov	O
(	O
2018	O
)	O
.	O
In	O
particular	O
,	O
we	O
test	O
the	O
proposed	O
ED	O
models	O
using	O
six	O
standard	O
datasets	O
:	O
AIDA	O
-	O
CoNLL	O
(	O
CoNLL	O
)	O
(	O
Hoffart	O
et	O
al	O
.	O
,	O
2011	O
)	O
,	O
MSNBC	O
,	O
AQUAINT	O
,	O
ACE2004	O
,	O
WNED	O
-	O
CWEB	O
(	O
CWEB	O
)	O
,	O
and	O
WNED	O
-	O
WIKI	O
(	O
WIKI	O
)	O
(	O
Guo	O
and	O
Barbosa	O
,	O
2018	O
)	O
.	O
We	O
consider	O
only	O
the	O
mentions	O
that	O
refer	O
to	O
valid	O
entities	O
in	O
Wikipedia	O
.	O
For	O
all	O
datasets	O
,	O
we	O
use	O
the	O
KB+YAGO	O
entity	O
candidates	O
and	O
their	O
associatedp(e|m	O
)	O
(	O
Ganea	O
and	O
Hofmann	O
,	O
2017	O
)	O
,	O
and	O
use	O
the	O
top	O
30	O
candidates	O
based	O
onp(e|m	O
)	O
.	O

For	O
the	O
CoNLL	O
dataset	O
,	O
we	O
also	O
test	O
the	O
performance	O
using	O
PPRforNED	O
entity	O
candidates	O
(	O
Pershina	O
et	O
al	O
.	O
,	O
2015	O
)	O
.	O
We	O
report	O
the	O
in	O
-	O
KB	O
accuracy	O
for	O
the	O
CoNLL	O
dataset	O
and	O
the	O
micro	O
F1	O
score	O
(	O
averaged	O
per	O
mention	O
)	O
for	O
the	O
other	O
datasets	O
.	O
Further	O
details	O
of	O
the	O
datasets	O
are	O
provided	O
in	O
Appendix	O
C.	O
Furthermore	O
,	O
we	O
optionally	O
fine	O
-	O
tune	O
the	O
model	O
by	O
maximizing	O
the	O
log	O
likelihood	O
of	O
the	O
ED	O
predictions	O
(	O
ŷ	O
ED	O
)	O
using	O
the	O
training	O
set	O
of	O
the	O
CoNLL	O
dataset	O
with	O
the	O
KB+YAGO	O
candidates	O
.	O
We	O
mask	O
90	O
%	O
of	O
the	O
mentions	O
and	O
fix	O
the	O
entity	O
token	O
embeddings	O
(	O
B	O
and	O
B	O
*	O
)	O
and	O
the	O
bias	O
(	O
The	O
model	O
is	O
trained	O
for	O
two	O
epochs	O
using	O
AdamW.	O
Additional	O
details	O
are	O
provided	O
in	O
Appendix	O
B.	O
Our	O
global	O
models	O
consistently	O
perform	O
better	O
than	O
the	O
local	O
model	O
,	O
demonstrating	O
the	O
effectiveness	O
of	O
using	O
global	O
contextual	O
information	O
even	O
if	O
local	O
contextual	O
information	O
is	O
captured	O
using	O
expressive	O
BERT	O
model	O
.	O
Moreover	O
,	O
the	O
confidenceorder	O
model	O
performs	O
better	O
than	O
the	O
natural	O
-	O
order	O
model	O
on	O
most	O
datasets	O
.	O
An	O
analysis	O
investigating	O
why	O
the	O
confidence	O
-	O
order	O
model	O
outperforms	O
the	O
natural	O
-	O
order	O
model	O
is	O
provided	O
in	O
the	O
next	O
section	O
.	O

Results	O

The	O
fine	O
-	O
tuning	O
on	O
the	O
CoNLL	O
dataset	O
significantly	O
improves	O
the	O
performance	O
on	O
this	O
dataset	O
(	O
Table	O
1	O
)	O
.	O
However	O
,	O
it	O
generally	O
degrades	O
the	O
performance	O
on	O
the	O
other	O
datasets	O
(	O
Table	O
2	O
)	O
.	O
This	O
suggests	O
that	O
Wikipedia	O
entity	O
annotations	O
are	O
more	O
suitable	O
than	O
the	O
CoNLL	O
dataset	O
to	O
train	O
generalpurpose	O
ED	O
models	O
.	O

Additionally	O
,	O
our	O
models	O
perform	O
worse	O
than	O
Yang	O
et	O
al	O
.	O
(	O
2018	O
)	O
on	O
the	O
CWEB	O
dataset	O
.	O
This	O
is	O
because	O
this	O
dataset	O
is	O
significantly	O
longer	O
on	O
average	O
than	O
other	O
datasets	O
,	O
i.e.	O
,	O
approximately	O
1,700	O
words	O
per	O
document	O
on	O
average	O
,	O
which	O
is	O
more	O
than	O
three	O
times	O
longer	O
than	O
the	O
512	O
-	O
word	O
limit	O
that	O
can	O
be	O
handled	O
by	O
BERT	O
-	O
based	O
models	O
including	O
ours	O
.	O
Yang	O
et	O
al	O
.	O
(	O
2018	O
)	O
achieved	O
excellent	O
performance	O
on	O
this	O
dataset	O
because	O
their	O
model	O
uses	O
various	O
hand	O
-	O
engineered	O
features	O
capturing	O
document	O
-	O
level	O
contextual	O
information	O
.	O

Analysis	O

To	O
investigate	O
how	O
global	O
contextual	O
information	O
helps	O
our	O
model	O
to	O
improve	O
performance	O
,	O
we	O
manually	O
analyze	O
the	O
difference	O
between	O
the	O
predictions	O
of	O
the	O
local	O
,	O
natural	O
-	O
order	O
,	O
and	O
confidence	O
-	O
order	O
models	O
.	O
We	O
use	O
the	O
fine	O
-	O
tuned	O
model	O
using	O
the	O
CoNLL	O
dataset	O
with	O
the	O
YAGO+KB	O
candidates	O
.	O
Although	O
all	O
models	O
perform	O
well	O
on	O
most	O
mentions	O
,	O
the	O
local	O
model	O
often	O
fails	O
to	O
resolve	O
mentions	O
of	O
common	O
names	O
referring	O
to	O
specific	O
entities	O
(	O
e.g.	O
,	O
"	O
New	O
York	O
"	O
referring	O
to	O
New	O
York	O
Knicks	O
)	O
.	O
Global	O
models	O
are	O
generally	O
better	O
to	O
resolve	O
such	O
difficult	O
cases	O
because	O
of	O
the	O
presence	O
of	O
strong	O
global	O
contextual	O
information	O
(	O
e.g.	O
,	O
mentions	O
refer	O
-	O
ring	O
to	O
basketball	O
teams	O
)	O
.	O

Furthermore	O
,	O
we	O
find	O
that	O
the	O
confidence	O
-	O
order	O
model	O
works	O
especially	O
well	O
for	O
mentions	O
that	O
require	O
a	O
highly	O
detailed	O
context	O
to	O
resolve	O
.	O
For	O
example	O
,	O
a	O
mention	O
of	O
"	O
Matthew	O
Burke	O
"	O
can	O
refer	O
to	O
two	O
different	O
former	O
Australian	O
rugby	O
players	O
.	O
Although	O
the	O
local	O
and	O
natural	O
-	O
order	O
models	O
incorrectly	O
resolve	O
this	O
mention	O
to	O
the	O
player	O
who	O
has	O
the	O
larger	O
number	O
of	O
occurrences	O
in	O
our	O
Wikipediabased	O
corpus	O
,	O
the	O
confidence	O
-	O
order	O
model	O
successfully	O
resolves	O
this	O
by	O
disambiguating	O
its	O
contextual	O
mentions	O
,	O
including	O
his	O
teammates	O
,	O
in	O
advance	O
.	O
We	O
provide	O
detailed	O
inference	O
sequence	O
of	O
the	O
corresponding	O
document	O
in	O
Appendix	O
D.	O

Performance	O
for	O
Rare	O
Entities	O

We	O
examine	O
whether	O
our	O
model	O
learns	O
effective	O
embeddings	O
for	O
rare	O
entities	O
using	O
the	O
CoNLL	O
dataset	O
.	O
Following	O
Ganea	O
and	O
Hofmann	O
(	O
2017	O
)	O
,	O
we	O
use	O
the	O
mentions	O
of	O
which	O
entity	O
candidates	O
contain	O
their	O
gold	O
entities	O
and	O
measure	O
the	O
performance	O
by	O
dividing	O
the	O
mentions	O
based	O
on	O
the	O
frequency	O
of	O
their	O
entities	O
in	O
the	O
Wikipedia	O
annotations	O
used	O
to	O
train	O
the	O
embeddings	O
.	O

As	O
presented	O
in	O
Table	O
3	O
,	O
our	O
models	O
achieve	O
enhanced	O
performance	O
for	O
rare	O
entities	O
.	O
Furthermore	O
,	O
the	O
global	O
models	O
consistently	O
outperform	O
the	O
local	O
model	O
both	O
for	O
rare	O
and	O
frequent	O
entities	O
.	O

Conclusion	O
and	O
Future	O
Work	O

We	O
propose	O
a	O
new	O
global	O
ED	O
model	O
based	O
on	O
BERT	O
.	O

Our	O
extensive	O
experiments	O
on	O
a	O
wide	O
range	O
of	O
ED	O
datasets	O
demonstrate	O
its	O
effectiveness	O
.	O

One	O
limitation	O
of	O
our	O
model	O
is	O
that	O
,	O
similar	O
to	O
existing	O
ED	O
models	O
,	O
our	O
model	O
can	O
not	O
handle	O
entities	O
that	O
are	O
not	O
included	O
in	O
the	O
vocabulary	O
.	O
In	O
our	O
future	O
work	O
,	O
we	O
will	O
investigate	O
the	O
method	O
to	O
compute	O
the	O
embeddings	O
of	O
such	O
entities	O
using	O
a	O
post	O
-	O
hoc	O
training	O
with	O
an	O
extended	O
vocabulary	O
(	O
Tai	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

Appendix	O
for	O
"	O
Global	O
Entity	O
Disambiguation	O
with	O
BERT	O
"	O
A	O
Details	O
of	O
Proposed	O
Model	O

As	O
the	O
input	O
corpus	O
for	O
training	O
our	O
model	O
,	O
we	O
use	O
the	O
December	O
2018	O
version	O
of	O
Wikipedia	O
,	O
comprising	O
approximately	O
3.5	O
billion	O
words	O
and	O
11	O
million	O
entity	O
annotations	O
.	O
We	O
generate	O
input	O
sequences	O
by	O
splitting	O
the	O
content	O
of	O
each	O
page	O
into	O
sequences	O
comprising	O
≤	O
512	O
words	O
and	O
their	O
entity	O
annotations	O
(	O
i.e.	O
,	O
hyperlinks	O
)	O
.	O
The	O
input	O
text	O
is	O
tokenized	O
using	O
BERT	O
's	O
tokenizer	O
with	O
its	O
vocabulary	O
consisting	O
of	O
V	O
w	O
=	O
30	O
,	O
000	O
words	O
.	O
Similar	O
to	O
Ganea	O
and	O
Hofmann	O
(	O
2017	O
)	O
,	O
we	O
create	O
an	O
entity	O
vocabulary	O
consisting	O
of	O
V	O
e	O
=	O
128	O
,	O
040	O
entities	O
,	O
which	O
are	O
contained	O
in	O
the	O
entity	O
candidates	O
in	O
the	O
datasets	O
used	O
in	O
our	O
experiments	O
.	O

Our	O
model	O
consists	O
of	O
approximately	O
440	O
million	O
parameters	O
.	O
To	O
reduce	O
the	O
training	O
time	O
,	O
the	O
parameters	O
that	O
are	O
shared	O
with	O
BERT	O
are	O
initialized	O
using	O
BERT	O
.	O
The	O
other	O
parameters	O
are	O
initialized	O
randomly	O
.	O
The	O
model	O
is	O
trained	O
via	O
iterations	O
over	O
Wikipedia	O
pages	O
in	O
a	O
random	O
order	O
for	O
seven	O
epochs	O
.	O
To	O
stabilize	O
the	O
training	O
,	O
we	O
update	O
only	O
those	O
parameters	O
that	O
are	O
randomly	O
initialized	O
(	O
i.e.	O
,	O
fixed	O
the	O
parameters	O
initialized	O
using	O
BERT	O
)	O
at	O
the	O
first	O
epoch	O
,	O
and	O
update	O
all	O
parameters	O
in	O
the	O
remaining	O
six	O
epochs	O
.	O
We	O
implement	O
the	O
model	O
using	O
PyTorch	O
(	O
Paszke	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
Hugging	O
Face	O
Transformers	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
and	O
the	O
training	O
takes	O
approximately	O
ten	O
days	O
using	O
eight	O
Tesla	O
V100	O
GPUs	O
.	O
We	O
optimize	O
the	O
model	O
using	O
AdamW.	O
The	O
hyper	O
-	O
parameters	O
used	O
in	O
the	O
training	O
are	O
detailed	O
in	O
Table	O
4	O
.	O

B	O
Details	O
of	O
Fine	O
-	O
tuning	O
on	O
CoNLL	O
Dataset	O

The	O
hyper	O
-	O
parameters	O
used	O
in	O
the	O
fine	O
-	O
tuning	O
on	O
the	O
CoNLL	O
dataset	O
are	O
detailed	O
in	O
Table	O
5	O
.	O
We	O
select	O
these	O
hyper	O
-	O
parameters	O
from	O
the	O
search	O
space	O
described	O
in	O
Devlin	O
et	O
al	O
.	O
(	O
2019	O
)	O
based	O
on	O
the	O
accuracy	O
on	O
the	O
development	O
set	O
of	O
the	O
CoNLL	O
dataset	O
.	O

A	O
document	O
is	O
split	O
if	O
it	O
is	O
longer	O
than	O
512	O
words	O
,	O
which	O
is	O
the	O
maximum	O
word	O
length	O
of	O
the	O
BERT	O
model	O
.	O

C	O
Details	O
of	O
ED	O
Datasets	O

The	O
statistics	O
of	O
the	O
ED	O
datasets	O
used	O
in	O
our	O
experiments	O
are	O
provided	O
in	O
Table	O
6	O
.	O

D	O
Example	O
of	O
Inference	O
by	O

Confidence	O
-	O
order	O
Model	O
Figure	O
3	O
shows	O
an	O
example	O
of	O
the	O
inference	O
performed	O
by	O
our	O
confidence	O
-	O
order	O
model	O
fine	O
-	O
tuned	O
on	O
the	O
CoNLL	O
dataset	O
.	O
The	O
document	O
is	O
obtained	O
from	O
the	O
test	O
set	O
of	O
the	O
CoNLL	O
dataset	O
.	O
As	O
shown	O
in	O
the	O
figure	O
,	O
the	O
model	O
starts	O
with	O
unambiguous	O
player	O
names	O
to	O
recognize	O
the	O
topic	O
of	O
the	O
document	O
,	O
and	O
subsequently	O
resolves	O
the	O
mentions	O
that	O
are	O
challenging	O
to	O
resolve	O
.	O
Notably	O
,	O
the	O
model	O
correctly	O
resolves	O
the	O
mention	O
"	O
Nigel	O
Walker	O
"	O
to	O
the	O
corresponding	O
former	O
rugby	O
player	O
instead	O
of	O
a	O
football	O
player	O
,	O
and	O
the	O
mention	O
"	O
Matthew	O
Burke	O
"	O
to	O
the	O
correct	O
former	O
Australian	O
rugby	O
player	O
born	O
in	O
1973	O
instead	O
of	O
the	O
former	O
Australian	O
rugby	O
player	O
born	O
in	O
1964	O
.	O
This	O
is	O
accomplished	O
by	O
resolving	O
other	O
contextual	O
mentions	O
,	O
including	O
their	O
colleague	O
players	O
,	O
in	O
advance	O
.	O
These	O
two	O
mentions	O
are	O
denoted	O
in	O
red	O
in	O
the	O
figure	O
.	O
Note	O
that	O
our	O
local	O
model	O
fails	O
to	O
resolve	O
both	O
mentions	O
,	O
and	O
our	O
natural	O
-	O
order	O
model	O
fails	O
to	O
resolve	O
"	O
Matthew	O
Burke	O
.	O
"	O

A	O
Matter	O
of	O
Framing	O
:	O
The	O
Impact	O
of	O
Linguistic	O
Formalism	O
on	O
Probing	O
Results	O

Deep	O
pre	O
-	O
trained	O
contextualized	O
encoders	O
like	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
demonstrate	O
remarkable	O
performance	O
on	O
a	O
range	O
of	O
downstream	O
tasks	O
.	O
A	O
recent	O
line	O
of	O
research	O
in	O
probing	O
investigates	O
the	O
linguistic	O
knowledge	O
implicitly	O
learned	O
by	O
these	O
models	O
during	O
pretraining	O
.	O
While	O
most	O
work	O
in	O
probing	O
operates	O
on	O
the	O
task	O
level	O
,	O
linguistic	O
tasks	O
are	O
rarely	O
uniform	O
and	O
can	O
be	O
represented	O
in	O
a	O
variety	O
of	O
formalisms	O
.	O
Any	O
linguistics	O
-	O
based	O
probing	O
study	O
thereby	O
inevitably	O
commits	O
to	O
the	O
formalism	O
used	O
to	O
annotate	O
the	O
underlying	O
data	O
.	O
Can	O
the	O
choice	O
of	O
formalism	O
affect	O
probing	O
results	O
?	O
To	O
investigate	O
,	O
we	O
conduct	O
an	O
in	O
-	O
depth	O
cross	O
-	O
formalism	O
layer	O
probing	O
study	O
in	O
role	O
semantics	O
.	O
We	O
find	O
linguistically	O
meaningful	O
differences	O
in	O
the	O
encoding	O
of	O
semantic	O
role	O
-	O
and	O
proto	O
-	O
role	O
information	O
by	O
BERT	O
depending	O
on	O
the	O
formalism	O
and	O
demonstrate	O
that	O
layer	O
probing	O
can	O
detect	O
subtle	O
differences	O
between	O
the	O
implementations	O
of	O
the	O
same	O
linguistic	O
formalism	O
.	O
Our	O
results	O
suggest	O
that	O
linguistic	O
formalism	O
is	O
an	O
important	O
dimension	O
in	O
probing	O
studies	O
and	O
should	O
be	O
investigated	O
along	O
with	O
the	O
commonly	O
used	O
cross	O
-	O
task	O
and	O
cross	O
-	O
lingual	O
experimental	O
settings	O
.	O

Introduction	O

The	O
emergence	O
of	O
deep	O
pre	O
-	O
trained	O
contextualized	O
encoders	O
has	O
had	O
a	O
major	O
impact	O
on	O
the	O
field	O
of	O
natural	O
language	O
processing	O
.	O
Boosted	O
by	O
the	O
availability	O
of	O
general	O
-	O
purpose	O
frameworks	O
like	O
AllenNLP	O
and	O
Transformers	O
(	O
Wolf	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
pre	O
-	O
trained	O
models	O
like	O
ELMO	O
and	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
have	O
caused	O
a	O
shift	O
towards	O
simple	O
architectures	O
where	O
a	O
strong	O
pre	O
-	O
trained	O
encoder	O
is	O
paired	O
with	O
a	O
shallow	O
downstream	O
model	O
,	O
often	O
outperforming	O
the	O
intricate	O
task	O
-	O
specific	O
architectures	O
of	O
the	O
past	O
.	O

The	O
versatility	O
of	O
pre	O
-	O
trained	O
representations	O
implies	O
that	O
they	O
encode	O
some	O
aspects	O
of	O
general	O

L=0	O
L=8	O
L=11	O

Figure	O
1	O
:	O
Intra	O
-	O
sentence	O
similarity	O
by	O
layer	O
L	O
of	O
the	O
multilingual	O
BERT	O
-	O
base	O
.	O
Functional	O
tokens	O
are	O
similar	O
in	O
L	O
=	O
0	O
,	O
syntactic	O
groups	O
emerge	O
at	O
higher	O
levels	O
.	O

linguistic	O
knowledge	O
(	O
Reif	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Indeed	O
,	O
even	O
an	O
informal	O
inspection	O
of	O
layer	O
-	O
wise	O
intrasentence	O
similarities	O
(	O
Fig	O
.	O
1	O
)	O
suggests	O
that	O
these	O
models	O
capture	O
elements	O
of	O
linguistic	O
structure	O
,	O
and	O
those	O
differ	O
depending	O
on	O
the	O
layer	O
of	O
the	O
model	O
.	O
A	O
grounded	O
investigation	O
of	O
these	O
regularities	O
allows	O
to	O
interpret	O
the	O
model	O
's	O
behaviour	O
,	O
design	O
better	O
pre	O
-	O
trained	O
encoders	O
and	O
inform	O
the	O
downstream	O
model	O
development	O
.	O
Such	O
investigation	O
is	O
the	O
main	O
subject	O
of	O
probing	O
,	O
and	O
recent	O
studies	O
confirm	O
that	O
BERT	O
implicitly	O
captures	O
many	O
aspects	O
of	O
language	O
use	O
,	O
lexical	O
semantics	O
and	O
grammar	O
(	O
Rogers	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

Most	O
probing	O
studies	O
use	O
linguistics	O
as	O
a	O
theoretical	O
scaffolding	O
and	O
operate	O
on	O
a	O
task	O
level	O
.	O
However	O
,	O
there	O
often	O
exist	O
multiple	O
ways	O
to	O
represent	O
the	O
same	O
linguistic	O
phenomenon	O
:	O
for	O
example	O
,	O
English	O
dependency	O
syntax	O
can	O
be	O
encoded	O
using	O
a	O
variety	O
of	O
formalisms	O
,	O
incl	O
.	O
Universal	O
(	O
Schuster	O
and	O
Manning	O
,	O
2016	O
)	O
,	O
Stanford	O
(	O
de	O
Marneffe	O
and	O
Manning	O
,	O
2008	O
)	O
and	O
CoNLL-2009	O
dependencies	O
(	O
Hajič	O
et	O
al	O
.	O
,	O
2009	O
)	O
,	O
all	O
using	O
different	O
label	O
sets	O
and	O
syntactic	O
head	O
attachment	O
rules	O
.	O
Any	O
probing	O
study	O
inevitably	O
commits	O
to	O
the	O
specific	O
theoretical	O
framework	O
used	O
to	O
produce	O
the	O
underlying	O
data	O
.	O
The	O
differences	O
between	O
linguistic	O
formalisms	O
,	O
however	O
,	O
can	O
be	O
substantial	O
.	O

Can	O
these	O
differences	O
affect	O
the	O
probing	O
results	O
?	O
This	O
question	O
is	O
intriguing	O
for	O
several	O
reasons	O
.	O
Lin	O
-	O
guistic	O
formalisms	O
are	O
well	O
-	O
documented	O
,	O
and	O
if	O
the	O
choice	O
of	O
formalism	O
indeed	O
has	O
an	O
effect	O
on	O
probing	O
,	O
cross	O
-	O
formalism	O
comparison	O
will	O
yield	O
new	O
insights	O
into	O
the	O
linguistic	O
knowledge	O
obtained	O
by	O
contextualized	O
encoders	O
during	O
pre	O
-	O
training	O
.	O
If	O
,	O
alternatively	O
,	O
the	O
probing	O
results	O
remain	O
stable	O
despite	O
substantial	O
differences	O
between	O
formalisms	O
,	O
this	O
prompts	O
a	O
further	O
scrutiny	O
of	O
what	O
the	O
pretrained	O
encoders	O
in	O
fact	O
encode	O
.	O
Finally	O
,	O
on	O
the	O
reverse	O
side	O
,	O
cross	O
-	O
formalism	O
probing	O
might	O
be	O
used	O
as	O
a	O
tool	O
to	O
empirically	O
compare	O
the	O
formalisms	O
and	O
their	O
language	O
-	O
specific	O
implementations	O
.	O
To	O
the	O
best	O
of	O
our	O
knowledge	O
we	O
are	O
the	O
first	O
to	O
explicitly	O
address	O
the	O
influence	O
of	O
formalism	O
on	O
probing	O
.	O

Ideally	O
,	O
the	O
task	O
chosen	O
for	O
a	O
cross	O
-	O
formalism	O
study	O
should	O
be	O
encoded	O
in	O
multiple	O
formalisms	O
using	O
the	O
same	O
textual	O
data	O
to	O
rule	O
out	O
the	O
influence	O
of	O
the	O
domain	O
and	O
text	O
type	O
.	O
While	O
many	O
linguistic	O
corpora	O
contain	O
several	O
layers	O
of	O
linguistic	O
information	O
,	O
having	O
the	O
same	O
textual	O
data	O
annotated	O
with	O
multiple	O
formalisms	O
for	O
the	O
same	O
task	O
is	O
rare	O
.	O
We	O
focus	O
on	O
role	O
semantics	O
-a	O
family	O
of	O
shallow	O
semantic	O
formalisms	O
at	O
the	O
interface	O
between	O
syntax	O
and	O
propositional	O
semantics	O
that	O
assign	O
roles	O
to	O
the	O
participants	O
of	O
natural	O
language	O
utterances	O
,	O
determining	O
who	O
did	O
what	O
to	O
whom	O
,	O
where	O
,	O
when	O
etc	O
.	O
Decades	O
of	O
research	O
in	O
theoretical	O
linguistics	O
have	O
produced	O
a	O
range	O
of	O
rolesemantic	O
frameworks	O
that	O
have	O
been	O
operationalized	O
in	O
NLP	O
:	O
syntax	O
-	O
driven	O
PropBank	O
(	O
Palmer	O
et	O
al	O
.	O
,	O
2005	O
)	O
,	O
coarse	O
-	O
grained	O
VerbNet	O
(	O
Kipper	O
-	O
Schuler	O
,	O
2005	O
)	O
,	O
fine	O
-	O
grained	O
FrameNet	O
(	O
Baker	O
et	O
al	O
.	O
,	O
1998	O
)	O
,	O
and	O
,	O
recently	O
,	O
decompositional	O
Semantic	O
Proto	O
-	O
Roles	O
(	O
SPR	O
)	O
(	O
Reisinger	O
et	O
al	O
.	O
,	O
2015;White	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
The	O
SemLink	O
project	O
(	O
Bonial	O
et	O
al	O
.	O
,	O
2013	O
)	O
offers	O
parallel	O
annotation	O
for	O
PropBank	O
,	O
VerbNet	O
and	O
FrameNet	O
for	O
English	O
.	O
This	O
allows	O
us	O
to	O
isolate	O
the	O
object	O
of	O
our	O
study	O
:	O
apart	O
from	O
the	O
rolesemantic	O
labels	O
,	O
the	O
underlying	O
data	O
and	O
conditions	O
for	O
the	O
three	O
formalisms	O
are	O
identical	O
.	O
SR3DE	O
(	O
Mújdricza	O
-	O
Maydt	O
et	O
al	O
.	O
,	O
2016	O
)	O
provides	O
compatible	O
annotation	O
in	O
three	O
formalisms	O
for	O
German	O
,	O
enabling	O
cross	O
-	O
lingual	O
validation	O
of	O
our	O
results	O
.	O
Combined	O
,	O
these	O
factors	O
make	O
role	O
semantics	O
an	O
ideal	O
target	O
for	O
our	O
cross	O
-	O
formalism	O
probing	O
study	O
.	O

A	O
solid	O
body	O
of	O
evidence	O
suggests	O
that	O
encoders	O
like	O
BERT	O
capture	O
syntactic	O
and	O
lexical	O
-	O
semantic	O
properties	O
,	O
but	O
only	O
few	O
studies	O
have	O
considered	O
probing	O
for	O
predicate	O
-	O
level	O
semantics	O
(	O
Tenney	O
et	O
al	O
.	O
,	O
2019b;Kovaleva	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
To	O
the	O
best	O
of	O
our	O
knowledge	O
we	O
are	O
the	O
first	O
to	O
conduct	O
a	O
cross	O
-	O
formalism	O
probing	O
study	O
on	O
role	O
semantics	O
,	O
thereby	O
contributing	O
to	O
the	O
line	O
of	O
research	O
on	O
how	O
and	O
whether	O
pre	O
-	O
trained	O
BERT	O
encodes	O
higher	O
-	O
level	O
semantic	O
phenomena	O
.	O

Contributions	O
.	O
This	O
work	O
studies	O
the	O
effect	O
of	O
the	O
linguistic	O
formalism	O
on	O
probing	O
results	O
.	O
We	O
conduct	O
cross	O
-	O
formalism	O
experiments	O
on	O
PropBank	O
,	O
VerbNet	O
and	O
FrameNet	O
role	O
prediction	O
in	O
English	O
and	O
German	O
,	O
and	O
show	O
that	O
the	O
formalism	O
can	O
affect	O
probing	O
results	O
in	O
a	O
linguistically	O
meaningful	O
way	O
;	O
in	O
addition	O
,	O
we	O
demonstrate	O
that	O
layer	O
probing	O
can	O
detect	O
subtle	O
differences	O
between	O
implementations	O
of	O
the	O
same	O
formalism	O
in	O
different	O
languages	O
.	O
On	O
the	O
technical	O
side	O
,	O
we	O
advance	O
the	O
recently	O
introduced	O
edge	O
and	O
layer	O
probing	O
framework	O
(	O
Tenney	O
et	O
al	O
.	O
,	O
2019b	O
)	O
;	O
in	O
particular	O
,	O
we	O
introduce	O
anchor	O
tasks	O
-an	O
analytical	O
tool	O
inspired	O
by	O
feature	O
-	O
based	O
systems	O
that	O
allows	O
deeper	O
qualitative	O
insights	O
into	O
the	O
pre	O
-	O
trained	O
models	O
'	O
behaviour	O
.	O
Finally	O
,	O
advancing	O
the	O
current	O
knowledge	O
about	O
the	O
encoding	O
of	O
predicate	O
semantics	O
in	O
BERT	O
,	O
we	O
perform	O
a	O
fine	O
-	O
grained	O
semantic	O
proto	O
-	O
role	O
probing	O
study	O
and	O
demonstrate	O
that	O
semantic	O
proto	O
-	O
role	O
properties	O
can	O
be	O
extracted	O
from	O
pre	O
-	O
trained	O
BERT	O
,	O
contrary	O
to	O
the	O
existing	O
reports	O
.	O
Our	O
results	O
suggest	O
that	O
along	O
with	O
task	O
and	O
language	O
,	O
linguistic	O
formalism	O
is	O
an	O
important	O
dimension	O
to	O
be	O
accounted	O
for	O
in	O
probing	O
research	O
.	O

Related	O
Work	O

BERT	O
as	O
Encoder	O

BERT	O
is	O
a	O
Transformer	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
encoder	O
pre	O
-	O
trained	O
by	O
jointly	O
optimizing	O
two	O
unsupervised	O
objectives	O
:	O
masked	O
language	O
model	O
and	O
next	O
sentence	O
prediction	O
.	O
It	O
uses	O
WordPiece	O
(	O
WP	O
,	O
Wu	O
et	O
al	O
.	O
(	O
2016	O
)	O
)	O
subword	O
tokens	O
along	O
with	O
positional	O
embeddings	O
as	O
input	O
,	O
and	O
gradually	O
constructs	O
sentence	O
representations	O
by	O
applying	O
tokenlevel	O
self	O
-	O
attention	O
pooling	O
over	O
a	O
stack	O
of	O
layers	O
L.	O
The	O
result	O
of	O
BERT	O
encoding	O
is	O
a	O
layer	O
-	O
wise	O
representation	O
of	O
the	O
input	O
wordpiece	O
tokens	O
with	O
higher	O
layers	O
representing	O
higher	O
-	O
level	O
abstractions	O
over	O
the	O
input	O
sequence	O
.	O
Thanks	O
to	O
the	O
joint	O
pre	O
-	O
training	O
objective	O
,	O
BERT	O
can	O
encode	O
words	O
and	O
sentences	O
in	O
a	O
unified	O
fashion	O
:	O
the	O
encoding	O
of	O
a	O
sentence	O
or	O
a	O
sentence	O
pair	O
is	O
stored	O
in	O
a	O
special	O
token	O
[	O
CLS	O
]	O
.	O

To	O
facilitate	O
multilingual	O
experiments	O
,	O
we	O
use	O
the	O
multilingual	O
BERT	O
-	O
base	O
(	O
mBERT	O
)	O
published	O
by	O
Devlin	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O
Although	O
several	O
recent	O
encoders	O
have	O
outperformed	O
BERT	O
on	O
benchmarks	O
Lan	O
et	O
al	O
.	O
,	O
2019;Raffel	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
we	O
use	O
the	O
original	O
BERT	O
architecture	O
,	O
since	O
it	O
allows	O
us	O
to	O
inherit	O
the	O
probing	O
methodology	O
and	O
to	O
build	O
upon	O
the	O
related	O
findings	O
.	O

Probing	O

Due	O
to	O
space	O
limitations	O
we	O
omit	O
high	O
-	O
level	O
discussions	O
on	O
benchmarking	O
(	O
Wang	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
sentence	O
-	O
level	O
probing	O
(	O
Conneau	O
et	O
al	O
.	O
,	O
2018a	O
)	O
,	O
and	O
focus	O
on	O
the	O
recent	O
findings	O
related	O
to	O
the	O
representation	O
of	O
linguistic	O
structure	O
in	O
BERT	O
.	O
Surface	O
-	O
level	O
information	O
generally	O
tends	O
to	O
be	O
represented	O
in	O
the	O
lower	O
layers	O
of	O
deep	O
encoders	O
,	O
while	O
higher	O
layers	O
store	O
hierarchical	O
and	O
semantic	O
information	O
(	O
Belinkov	O
et	O
al	O
.	O
,	O
2017;Lin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
show	O
that	O
the	O
abstraction	O
strategy	O
applied	O
by	O
the	O
English	O
pre	O
-	O
trained	O
BERT	O
encoder	O
follows	O
the	O
order	O
of	O
the	O
classical	O
NLP	O
pipeline	O
.	O
Strengthening	O
the	O
claim	O
about	O
linguistic	O
capabilities	O
of	O
BERT	O
,	O
Hewitt	O
and	O
Manning	O
(	O
2019	O
)	O
demonstrate	O
that	O
BERT	O
implicitly	O
learns	O
syntax	O
,	O
and	O
Reif	O
et	O
al	O
.	O
(	O
2019	O
)	O
show	O
that	O
it	O
encodes	O
fine	O
-	O
grained	O
lexicalsemantic	O
distinctions	O
.	O
Rogers	O
et	O
al	O
.	O
(	O
2020	O
)	O
provide	O
a	O
comprehensive	O
overview	O
of	O
BERT	O
's	O
properties	O
discovered	O
to	O
date	O
.	O

While	O
recent	O
results	O
indicate	O
that	O
BERT	O
successfully	O
represents	O
lexical	O
-	O
semantic	O
and	O
grammatical	O
information	O
,	O
the	O
evidence	O
of	O
its	O
high	O
-	O
level	O
semantic	O
capabilities	O
is	O
inconclusive	O
.	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
show	O
that	O
the	O
English	O
PropBank	O
semantics	O
can	O
be	O
extracted	O
from	O
the	O
encoder	O
and	O
follows	O
syntax	O
in	O
the	O
layer	O
structure	O
.	O
However	O
,	O
out	O
of	O
all	O
formalisms	O
PropBank	O
is	O
most	O
closely	O
tied	O
to	O
syntax	O
,	O
and	O
the	O
results	O
on	O
proto	O
-	O
role	O
and	O
relation	O
probing	O
do	O
not	O
follow	O
the	O
same	O
pattern	O
.	O
Kovaleva	O
et	O
al	O
.	O
(	O
2019	O
)	O
identify	O
two	O
attention	O
heads	O
in	O
BERT	O
responsible	O
for	O
FrameNet	O
relations	O
.	O
However	O
,	O
they	O
find	O
that	O
disabling	O
them	O
in	O
a	O
fine	O
-	O
tuning	O
evaluation	O
on	O
the	O
GLUE	O
(	O
Wang	O
et	O
al	O
.	O
,	O
2018	O
)	O
benchmark	O
does	O
not	O
result	O
in	O
decreased	O
performance	O
.	O

Although	O
we	O
are	O
not	O
aware	O
of	O
any	O
systematic	O
studies	O
dedicated	O
to	O
the	O
effect	O
of	O
formalism	O
on	O
probing	O
results	O
,	O
the	O
evidence	O
of	O
such	O
effects	O
is	O
scattered	O
across	O
the	O
related	O
work	O
:	O
for	O
example	O
,	O
the	O
aforementioned	O
results	O
in	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
show	O
a	O
difference	O
in	O
layer	O
utilization	O
between	O
constituents	O
-	O
and	O
dependency	O
-	O
based	O
syntactic	O
probes	O
and	O
semantic	O
role	O
and	O
proto	O
-	O
role	O
probes	O
.	O
It	O
is	O
not	O
clear	O
whether	O
this	O
effect	O
is	O
due	O
to	O
the	O
differences	O
in	O
the	O
underlying	O
datasets	O
and	O
task	O
architecture	O
or	O
the	O
formalism	O
per	O
se	O
.	O

Our	O
probing	O
methodology	O
builds	O
upon	O
the	O
edge	O
and	O
layer	O
probing	O
framework	O
.	O
The	O
encoding	O
produced	O
by	O
a	O
frozen	O
BERT	O
model	O
can	O
be	O
seen	O
as	O
a	O
layer	O
-	O
wise	O
snapshot	O
that	O
reflects	O
how	O
the	O
model	O
has	O
constructed	O
the	O
high	O
-	O
level	O
abstractions	O
.	O
Tenney	O
et	O
al	O
.	O
(	O
2019b	O
)	O
introduce	O
the	O
edge	O
probing	O
task	O
design	O
:	O
a	O
simple	O
classifier	O
is	O
tasked	O
with	O
predicting	O
a	O
linguistic	O
property	O
given	O
a	O
pair	O
of	O
spans	O
encoded	O
using	O
a	O
frozen	O
pre	O
-	O
trained	O
model	O
.	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
use	O
edge	O
probing	O
to	O
analyse	O
the	O
layer	O
utilization	O
of	O
a	O
pre	O
-	O
trained	O
BERT	O
model	O
via	O
scalar	O
mixing	O
weights	O
learned	O
during	O
training	O
.	O
We	O
revisit	O
this	O
framework	O
in	O
Section	O
3	O
.	O

Role	O
Semantics	O

We	O
now	O
turn	O
to	O
the	O
object	O
of	O
our	O
investigation	O
:	O
role	O
semantics	O
.	O
For	O
further	O
discussion	O
,	O
consider	O
the	O
following	O
synthetic	O
example	O
:	O

a.	O
Despite	O
surface	O
-	O
level	O
differences	O
,	O
the	O
sentences	O
express	O
the	O
same	O
meaning	O
,	O
suggesting	O
an	O
underlying	O
semantic	O
representation	O
in	O
which	O
these	O
sentences	O
are	O
equivalent	O
.	O
One	O
such	O
representation	O
is	O
offered	O
by	O
role	O
semantics	O
-a	O
shallow	O
predicatesemantic	O
formalism	O
closely	O
related	O
to	O
syntax	O
.	O
In	O
terms	O
of	O
role	O
semantics	O
,	O
Mary	O
,	O
book	O
and	O
John	O
are	O
semantic	O
arguments	O
of	O
the	O
predicate	O
give	O
,	O
and	O
are	O
assigned	O
roles	O
from	O
a	O
pre	O
-	O
defined	O
inventory	O
,	O
for	O
example	O
,	O
Agent	O
,	O
Recipient	O
and	O
Theme	O
.	O

Semantic	O
roles	O
and	O
their	O
properties	O
have	O
received	O
extensive	O
attention	O
in	O
linguistics	O
(	O
Fillmore	O
,	O
1968;Levin	O
and	O
Rappaport	O
Hovav	O
,	O
2005;Dowty	O
,	O
1991	O
)	O
and	O
are	O
considered	O
a	O
universal	O
feature	O
of	O
human	O
language	O
.	O
The	O
size	O
and	O
organization	O
of	O
the	O
role	O
and	O
predicate	O
inventory	O
are	O
subject	O
to	O
debate	O
,	O
giving	O
rise	O
to	O
a	O
variety	O
of	O
role	O
-	O
semantic	O
formalisms	O
.	O

PropBank	O
assumes	O
a	O
predicate	O
-	O
independent	O
labeling	O
scheme	O
where	O
predicates	O
are	O
distinguished	O
by	O
their	O
sense	O
(	O
get.01	O
)	O
,	O
and	O
semantic	O
arguments	O
are	O
labeled	O
with	O
generic	O
numbered	O
core	O
(	O
Arg0	O
-	O
5	O
)	O
and	O
modifier	O
(	O
e.g.	O
AM	O
-	O
TMP	O
)	O
roles	O
.	O
Core	O
roles	O
are	O
not	O
tied	O
to	O
specific	O
definitions	O
,	O
but	O
the	O
effort	O
has	O
been	O
made	O
to	O
keep	O
the	O
role	O
assignments	O
consistent	O
for	O
similar	O
verbs	O
;	O
Arg0	O
and	O
Arg1	O
correspond	O
to	O
the	O
Proto	O
-	O
Agent	O
and	O
Proto	O
-	O
Patient	O
roles	O
as	O
per	O
Dowty	O
(	O
1991	O
)	O
.	O
The	O
semantic	O
interpretation	O
of	O
core	O
roles	O
depends	O
on	O
the	O
predicate	O
sense	O
.	O

VerbNet	O
follows	O
a	O
different	O
categorization	O
scheme	O
.	O
Motivated	O
by	O
the	O
regularities	O
in	O
verb	O
behavior	O
,	O
Levin	O
(	O
1993	O
)	O
has	O
introduced	O
the	O
group	O
-	O
ing	O
of	O
verbs	O
into	O
intersective	O
classes	O
(	O
ILC	O
)	O
.	O
This	O
methodology	O
has	O
been	O
adopted	O
by	O
VerbNet	O
:	O
for	O
example	O
,	O
the	O
VerbNet	O
class	O
get-13.5.1	O
would	O
include	O
verbs	O
earn	O
,	O
fetch	O
,	O
gain	O
etc	O
.	O
A	O
verb	O
in	O
Verb	O
-	O
Net	O
can	O
belong	O
to	O
several	O
classes	O
corresponding	O
to	O
different	O
senses	O
;	O
each	O
class	O
is	O
associated	O
with	O
a	O
set	O
of	O
roles	O
and	O
licensed	O
syntactic	O
transformations	O
.	O
Unlike	O
PropBank	O
,	O
VerbNet	O
uses	O
a	O
set	O
of	O
approx	O
.	O
30	O
thematic	O
roles	O
that	O
have	O
universal	O
definitions	O
and	O
are	O
shared	O
among	O
predicates	O
,	O
e.g.	O
Agent	O
,	O
Beneficiary	O
,	O
Instrument	O
.	O

FrameNet	O
takes	O
a	O
meaning	O
-	O
driven	O
stance	O
on	O
the	O
role	O
encoding	O
by	O
modeling	O
it	O
in	O
terms	O
of	O
frame	O
semantics	O
:	O
predicates	O
are	O
grouped	O
into	O
frames	O
(	O
e.g.	O
Commerce	O
buy	O
)	O
,	O
which	O
specify	O
role	O
-	O
like	O
slots	O
to	O
be	O
filled	O
.	O
FrameNet	O
offers	O
fine	O
-	O
grained	O
frame	O
distinctions	O
,	O
and	O
roles	O
in	O
FrameNet	O
are	O
frame	O
-	O
specific	O
,	O
e.g.	O
Buyer	O
,	O
Seller	O
and	O
Money	O
.	O
The	O
resource	O
accompanies	O
each	O
frame	O
with	O
a	O
description	O
of	O
the	O
situation	O
and	O
its	O
core	O
and	O
peripheral	O
participants	O
.	O

SPR	O
follows	O
the	O
work	O
of	O
Dowty	O
(	O
1991	O
)	O
and	O
discards	O
the	O
notion	O
of	O
categorical	O
semantic	O
roles	O
in	O
favor	O
of	O
feature	O
bundles	O
.	O

Instead	O
of	O
a	O
fixed	O
role	O
label	O
,	O
each	O
argument	O
is	O
assessed	O
via	O
a	O
11	O
-	O
dimensional	O
cardinal	O
feature	O
set	O
including	O
Proto	O
-	O
Agent	O
and	O
Proto	O
-	O
Patient	O
properties	O
like	O
volitional	O
,	O
sentient	O
,	O
destroyed	O
,	O
etc	O
.	O
The	O
feature	O
-	O
based	O
approach	O
eliminates	O
some	O
of	O
the	O
theoretical	O
issues	O
associated	O
with	O
categorical	O
role	O
inventories	O
and	O
allows	O
for	O
more	O
flexible	O
modeling	O
of	O
role	O
semantics	O
.	O

Each	O
of	O
the	O
role	O
labeling	O
formalisms	O
offers	O
certain	O
advantages	O
and	O
disadvantages	O
(	O
Giuglea	O
and	O
Moschitti	O
,	O
2006;Mújdricza	O
-	O
Maydt	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
While	O
being	O
close	O
to	O
syntax	O
and	O
thereby	O
easier	O
to	O
predict	O
,	O
PropBank	O
does	O
n't	O
contribute	O
much	O
semantics	O
to	O
the	O
representation	O
.	O
On	O
the	O
opposite	O
side	O
of	O
the	O
spectrum	O
,	O
FrameNet	O
offers	O
rich	O
predicatesemantic	O
representations	O
for	O
verbs	O
and	O
nouns	O
,	O
but	O
suffers	O
from	O
high	O
granularity	O
and	O
coverage	O
gaps	O
(	O
Hartmann	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
VerbNet	O
takes	O
a	O
middle	O
ground	O
by	O
following	O
grammatical	O
criteria	O
while	O
still	O
encoding	O
coarse	O
-	O
grained	O
semantics	O
,	O
but	O
only	O
focuses	O
on	O
verbs	O
and	O
core	O
(	O
not	O
modifier	O
)	O
roles	O
.	O
SPR	O
avoids	O
the	O
granularity	O
-	O
generalization	O
trade	O
-	O
off	O
of	O
the	O
categorical	O
inventories	O
,	O
but	O
is	O
yet	O
to	O
find	O
its	O
way	O
into	O
practical	O
NLP	O
applications	O
.	O

Probing	O
Methodology	O

We	O
take	O
the	O
edge	O
probing	O
setup	O
by	O
Tenney	O
et	O
al	O
.	O
(	O
2019b	O
)	O
as	O
our	O
starting	O
point	O
.	O
Edge	O
probing	O
aims	O
to	O
predict	O
a	O
label	O
given	O
a	O
pair	O
of	O
contextualized	O
span	O
or	O
word	O
encodings	O
.	O
More	O
formally	O
,	O
we	O
encode	O
a	O
WP	O
-	O
tokenized	O
sentence	O
[	O
wp	O
1	O
,	O
wp	O
2	O
,	O
...	O
wp	O
k	O
]	O
with	O
a	O
frozen	O
pre	O
-	O
trained	O
model	O
,	O
producing	O
contextual	O
embeddings	O
[	O
e	O
1	O
,	O
e	O
2	O
,	O
...	O
e	O
k	O
]	O
,	O
each	O
of	O
which	O
is	O
a	O
layered	O
representation	O
over	O
L	O
=	O
{	O
l	O
0	O
,	O
l	O
1	O
,	O
...	O
l	O
m	O
}	O
layers	O
,	O
with	O
encoding	O
at	O
layer	O
l	O
n	O
for	O
the	O
wordpiece	O
wp	O
i	O
further	O
denoted	O
as	O
e	O
n	O
i	O
.	O
A	O
trainable	O
scalar	O
mix	O
is	O
applied	O
to	O
the	O
layered	O
representation	O
to	O
produce	O
the	O
final	O
encoding	O
given	O
the	O
per	O
-	O
layer	O
mixing	O
weights	O
{	O
a	O
0	O
,	O
a	O
1	O
..	O
a	O
m	O
}	O
and	O
a	O
scaling	O
parameter	O
γ	O
:	O

e	O
i	O
=	O
γ	O
m	O
l=0	O
sof	O
tmax(a	O
l	O
)	O
e	O
l	O
i	O

Given	O
the	O
source	O
src	O
and	O
target	O
tgt	O
wordpieces	O
encoded	O
as	O
e	O
src	O
and	O
e	O
tgt	O
,	O
our	O
goal	O
is	O
to	O
predict	O
the	O
label	O
y.	O

Due	O
to	O
its	O
task	O
-	O
agnostic	O
architecture	O
,	O
edge	O
probing	O
can	O
be	O
applied	O
to	O
a	O
wide	O
variety	O
of	O
unary	O
(	O
by	O
omitting	O
tgt	O
)	O
and	O
binary	O
labeling	O
tasks	O
in	O
a	O
unified	O
manner	O
,	O
facilitating	O
the	O
cross	O
-	O
task	O
comparison	O
.	O
The	O
original	O
setup	O
has	O
several	O
limitations	O
that	O
we	O
address	O
in	O
our	O
implementation	O
.	O

Regression	O
tasks	O
.	O
The	O
original	O
edge	O
probing	O
setup	O
only	O
considers	O
classification	O
tasks	O
.	O
Many	O
language	O
phenomena	O
-including	O
positional	O
information	O
and	O
semantic	O
proto	O
-	O
roles	O
,	O
are	O
naturally	O
modeled	O
as	O
regression	O
.	O
We	O
extend	O
the	O
architecture	O
by	O
Tenney	O
et	O
al	O
.	O
(	O
2019b	O
)	O
and	O
support	O
both	O
classification	O
and	O
regression	O
:	O
the	O
former	O
achieved	O
via	O
softmax	O
,	O
the	O
latter	O
via	O
direct	O
linear	O
regression	O
to	O
the	O
target	O
value	O
.	O

Flat	O
model	O
.	O
To	O
decrease	O
the	O
models	O
'	O
own	O
expressive	O
power	O
(	O
Hewitt	O
and	O
Liang	O
,	O
2019	O
)	O
,	O
we	O
keep	O
the	O
number	O
of	O
parameters	O
in	O
our	O
probing	O
model	O
as	O
low	O
as	O
possible	O
.	O
While	O
Tenney	O
et	O
al	O
.	O
(	O
2019b	O
)	O
utilize	O
pooled	O
self	O
-	O
attentional	O
span	O
representations	O
and	O
a	O
projection	O
layer	O
to	O
enable	O
cross	O
-	O
model	O
comparison	O
,	O
we	O
directly	O
feed	O
the	O
wordpiece	O
encoding	O
into	O
the	O
classifier	O
,	O
using	O
the	O
first	O
wordpiece	O
of	O
a	O
word	O
.	O
To	O
further	O
increase	O
the	O
selectivity	O
of	O
the	O
model	O
,	O
we	O
directly	O
project	O
the	O
source	O
and	O
target	O
wordpiece	O
representations	O
into	O
the	O
label	O
space	O
,	O
opposed	O
to	O
the	O
two	O
-	O
layer	O
MLP	O
classifier	O
used	O
in	O
the	O
original	O
setup	O
.	O

Separate	O
scalar	O
mixes	O
.	O
To	O
enable	O
fine	O
-	O
grained	O
analysis	O
of	O
probing	O
results	O
,	O
we	O
train	O
and	O
analyze	O
separate	O
scalar	O
mixes	O
for	O
source	O
and	O
target	O
wordpieces	O
,	O
motivated	O
by	O
the	O
fact	O
that	O
the	O
classifier	O
might	O
utilize	O
different	O
aspects	O
of	O
their	O
representation	O
for	O
prediction	O
1	O
.	O
Indeed	O
,	O
we	O
find	O
that	O
the	O
mixing	O
weights	O
learned	O
for	O
source	O
and	O
target	O
wordpieces	O
might	O
show	O
substantial	O
-and	O
linguistically	O
meaningful	O
-variation	O
.	O
Combined	O
with	O
regressionbased	O
objective	O
,	O
separating	O
the	O
scalar	O
mixes	O
allows	O
us	O
to	O
scrutinize	O
layer	O
utilization	O
patterns	O
for	O
semantic	O
proto	O
-	O
roles	O
.	O

Sentence	O
-	O
level	O
probes	O
.	O
Utilizing	O
the	O
BERTspecific	O
sentence	O
representation	O
[	O
CLS	O
]	O
allows	O
us	O
to	O
incorporate	O
the	O
sentence	O
-	O
level	O
natural	O
language	O
inference	O
(	O
NLI	O
)	O
probe	O
into	O
our	O
kit	O
.	O

Anchor	O
tasks	O
.	O
We	O
employ	O
two	O
analytical	O
tools	O
from	O
the	O
original	O
layer	O
probing	O
setup	O
.	O
Mixing	O
weight	O
plotting	O
compares	O
layer	O
utilization	O
among	O
tasks	O
by	O
visually	O
aligning	O
the	O
respective	O
learned	O
weight	O
distributions	O
transformed	O
via	O
a	O
softmax	O
function	O
.	O
Layer	O
center	O
-	O
of	O
-	O
gravity	O
is	O
used	O
as	O
a	O
summary	O
statistic	O
for	O
a	O
task	O
's	O
layer	O
utilization	O
.	O
While	O
the	O
distribution	O
of	O
mixing	O
weights	O
along	O
the	O
layers	O
allows	O
us	O
to	O
estimate	O
the	O
order	O
in	O
which	O
information	O
is	O
processed	O
during	O
encoding	O
,	O
it	O
does	O
n't	O
allow	O
to	O
directly	O
assess	O
the	O
similarity	O
between	O
the	O
layer	O
utilization	O
of	O
the	O
probing	O
tasks	O
.	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
have	O
demonstrated	O
that	O
the	O
order	O
in	O
which	O
linguistic	O
information	O
is	O
stored	O
in	O
BERT	O
mirrors	O
the	O
traditional	O
NLP	O
pipeline	O
.	O
A	O
prominent	O
property	O
of	O
the	O
NLP	O
pipelines	O
is	O
their	O
use	O
of	O
low	O
-	O
level	O
features	O
to	O
predict	O
downstream	O
phenomena	O
.	O
In	O
the	O
context	O
of	O
layer	O
probing	O
,	O
probing	O
tasks	O
can	O
be	O
seen	O
as	O
end	O
-	O
to	O
-	O
end	O
feature	O
extractors	O
.	O
Following	O
this	O
intuition	O
,	O
we	O
define	O
two	O
groups	O
of	O
probing	O
tasks	O
:	O
target	O
tasks	O
-the	O
main	O
tasks	O
under	O
investigation	O
,	O
and	O
anchor	O
tasks	O
-a	O
set	O
of	O
related	O
tasks	O
that	O
serve	O
as	O
a	O
basis	O
for	O
qualitative	O
comparison	O
between	O
the	O
targets	O
.	O
The	O
softmax	O
transformation	O
of	O
the	O
scalar	O
mixing	O
weights	O
allows	O
to	O
treat	O
them	O
as	O
probability	O
distributions	O
:	O
the	O
higher	O
the	O
mixing	O
weight	O
of	O
a	O
layer	O
,	O
the	O
more	O
likely	O
the	O
probe	O
is	O
to	O
utilize	O
information	O
from	O
this	O
layer	O
during	O
prediction	O
.	O
We	O
use	O
Kullback	O
-	O
Leibler	O
divergence	O
to	O
compare	O
target	O
tasks	O
(	O
e.g.	O
role	O
labeling	O
in	O
different	O
formalisms	O
)	O
in	O
terms	O
of	O
their	O
similarity	O
to	O
lowerlevel	O
anchor	O
tasks	O
(	O
e.g.	O
dependency	O
relation	O
and	O
lemma	O
)	O
.	O
Note	O
that	O
the	O
notion	O
of	O
anchor	O
task	O
is	O
contextual	O
:	O
the	O
same	O
task	O
can	O
serve	O
as	O
a	O
target	O
and	O
as	O
an	O
anchor	O
,	O
depending	O
on	O
the	O
focus	O
of	O
the	O
study	O
.	O
jections	O
in	O
the	O
background	O
,	O
but	O
do	O
not	O
investigate	O
the	O
differences	O
between	O
the	O
learned	O
projections	O
.	O

Probing	O
tasks	O

Our	O
probing	O
kit	O
spans	O
a	O
wide	O
range	O
of	O
probing	O
tasks	O
,	O
from	O
primitive	O
surface	O
-	O
level	O
tasks	O
mostly	O
utilized	O
as	O
anchors	O
later	O
to	O
high	O
-	O
level	O
semantic	O
tasks	O
that	O
language	O
en	O
de	O
aim	O
to	O
provide	O
a	O
representational	O
upper	O
bound	O
to	O
predicate	O
semantics	O
.	O
We	O
follow	O
the	O
training	O
,	O
test	O
and	O
development	O
splits	O
from	O
the	O
original	O
SR3de	O
,	O
CoNLL-2009	O
and	O
SPR	O
data	O
.	O
The	O
XNLI	O
task	O
is	O
sourced	O
from	O
the	O
development	O
set	O
and	O
only	O
used	O
for	O
scalar	O
mix	O
analysis	O
.	O
To	O
reduce	O
the	O
number	O
of	O
labels	O
in	O
some	O
of	O
the	O
probing	O
tasks	O
,	O
we	O
collect	O
frequency	O
statistics	O
over	O
the	O
corresponding	O
training	O
sets	O
and	O
only	O
consider	O
up	O
to	O
250	O
most	O
frequent	O
labels	O
.	O
Below	O
we	O
define	O
the	O
tasks	O
in	O
order	O
of	O
their	O
complexity	O
,	O
Table	O
2	O
provides	O
the	O
probing	O
task	O
statistics	O
,	O
Table	O
3	O
compares	O
the	O
categorical	O
role	O
labeling	O
formalisms	O
in	O
terms	O
of	O
granularity	O
,	O
and	O
Table	O
4	O
provides	O
examples	O
.	O
We	O
evaluate	O
the	O
classification	O
performance	O
using	O
Accuracy	O
,	O
while	O
regression	O
tasks	O
are	O
scored	O
via	O
R	O
2	O
.	O

Token	O
type	O
(	O
ttype	O
)	O
predicts	O
the	O
type	O
of	O
a	O
word	O
.	O
This	O
requires	O
contextual	O
processing	O
since	O
a	O
word	O
might	O
consist	O
of	O
several	O
wordpieces	O
;	O
Token	O
position	O
(	O
token.ix	O
)	O
predicts	O
the	O
linear	O
position	O
of	O
a	O
word	O
,	O
cast	O
as	O
a	O
regression	O
task	O
over	O
the	O
first	O
20	O
words	O
in	O
the	O
sentence	O
.	O
Again	O
,	O
the	O
task	O
is	O
non	O
-	O
trivial	O
since	O
it	O
requires	O
the	O
words	O
to	O
be	O
assembled	O
from	O
the	O
wordpieces	O
.	O
Part	O
-	O
of	O
-	O
speech	O
(	O
pos	O
)	O
predicts	O
the	O
languagespecific	O
part	O
-	O
of	O
-	O
speech	O
tag	O
for	O
the	O
given	O
token	O
.	O
Lexical	O
unit	O
(	O
lex.unit	O
)	O
predicts	O
the	O
lemma	O
and	O
POS	O
of	O
the	O
given	O
word	O
-a	O
common	O
input	O
representation	O
for	O
the	O
entries	O
in	O
lexical	O
resources	O
.	O
We	O
extract	O
coarse	O
POS	O
tags	O
by	O
using	O
the	O
first	O
character	O
of	O
the	O
language	O
-	O
specific	O
POS	O
tag	O
.	O

Dependency	O
relation	O
(	O
deprel	O
)	O
predicts	O
the	O
dependency	O
relation	O
between	O
the	O
parent	O
src	O
and	O
dependent	O
tgt	O
tokens	O
;	O
Semantic	O
role	O
(	O
role.[frm	O
]	O
)	O
predicts	O
the	O
semantic	O
role	O
given	O
a	O
predicate	O
src	O
and	O
an	O
argument	O
tgt	O
token	O
in	O
one	O
of	O
the	O
three	O
role	O
labeling	O
formalisms	O
:	O
PropBank	O
pb	O
,	O
VerbNet	O
vn	O
and	O
FrameNet	O
fn	O
.	O
Note	O
that	O
we	O
only	O
probe	O
for	O
the	O
role	O
label	O
,	O
and	O
the	O
model	O
has	O
no	O
access	O
to	O
the	O
verb	O
sense	O
information	O
from	O
the	O
data	O
.	O
Semantic	O
proto	O
-	O
role	O
(	O
spr	O
.	O
[	O
prop	O
]	O
)	O
is	O
a	O
set	O
of	O
eleven	O
regression	O
tasks	O
predicting	O
the	O
values	O
of	O
the	O
proto	O
-	O
role	O
properties	O
as	O
defined	O
in	O
(	O
Reisinger	O
et	O
al	O
.	O
,	O
2015	O
)	O
,	O
given	O
a	O
predicate	O
src	O
and	O
an	O
argument	O
tgt	O
.	O
XNLI	O
is	O
a	O
sentence	O
-	O
level	O
NLI	O
task	O
directly	O
sourced	O
from	O
the	O
corresponding	O
dataset	O
.	O
Given	O
two	O
sentences	O
,	O
the	O
goal	O
is	O
to	O
determine	O
whether	O
an	O
entailment	O
or	O
a	O
contradiction	O
relationship	O
holds	O
between	O
them	O
.	O
We	O
use	O
NLI	O
to	O
investigate	O
the	O
layer	O
utilization	O
of	O
mBERT	O
for	O
high	O
-	O
level	O
semantic	O
tasks	O
.	O
We	O
extract	O
the	O
sentence	O
pair	O
representation	O
via	O
the	O
[	O
CLS	O
]	O
token	O
and	O
treat	O
it	O
as	O
a	O
unary	O
probing	O
task	O
.	O

Results	O

Our	O
probing	O
framework	O
is	O
implemented	O
using	O
AllenNLP	O
.	O
2	O
We	O
train	O
the	O
probes	O
for	O
20	O
epochs	O
using	O
the	O
Adam	O
optimizer	O
with	O
default	O
parameters	O
and	O
a	O
batch	O
size	O
of	O
32	O
.	O
Due	O
to	O
the	O
frozen	O
encoder	O
and	O
flat	O
model	O
architecture	O
,	O
the	O
total	O
runtime	O
of	O
the	O
main	O
experiments	O
is	O
under	O
8	O
hours	O
on	O
a	O
single	O
Tesla	O
V100	O
GPU	O
.	O
In	O
addition	O
to	O
pre	O
-	O
trained	O
mBERT	O
we	O
report	O
baseline	O
performance	O
using	O
a	O
frozen	O
untrained	O
mBERT	O
model	O
obtained	O
by	O
randomizing	O
the	O
encoder	O
weights	O
post	O
-	O
initialization	O
as	O
in	O
Jawahar	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O

General	O
Trends	O

While	O
absolute	O
performance	O
is	O
secondary	O
to	O
our	O
analysis	O
,	O
we	O
report	O
the	O
probing	O
task	O
scores	O
on	O
respective	O
development	O
sets	O
in	O
Table	O
5	O
.	O
We	O
observe	O
that	O
grammatical	O
tasks	O
score	O
high	O
,	O
while	O
core	O
role	O
labeling	O
lags	O
behind	O
-in	O
line	O
with	O
the	O
findings	O
of	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
3	O
We	O
observe	O
lower	O
scores	O
for	O
German	O
role	O
labeling	O
which	O
we	O
attribute	O
to	O
the	O
lack	O
of	O
training	O
data	O
.	O
Surprisingly	O
,	O
as	O
we	O
show	O
below	O
,	O
this	O
does	O
n't	O
prevent	O
the	O
edge	O
probe	O
from	O
task	O
en	O
de	O
*	O
token.ix	O
0.95	O
(	O
0.93	O
)	O
0.92	O
(	O
0	O
.	O
learning	O
to	O
locate	O
relevant	O
role	O
-	O
semantic	O
information	O
in	O
mBERT	O
's	O
layers	O
.	O

The	O
untrained	O
mBERT	O
baseline	O
expectedly	O
underperforms	O
;	O
however	O
,	O
we	O
note	O
good	O
baseline	O
results	O
on	O
surface	O
-	O
level	O
tasks	O
for	O
English	O
,	O
which	O
we	O
attribute	O
to	O
memorizing	O
token	O
identity	O
and	O
position	O
:	O
although	O
the	O
weights	O
are	O
set	O
randomly	O
,	O
the	O
frozen	O
encoder	O
still	O
associates	O
each	O
wordpiece	O
input	O
with	O
a	O
fixed	O
random	O
vector	O
.	O
We	O
have	O
confirmed	O
this	O
assumption	O
by	O
scalar	O
mix	O
analysis	O
of	O
the	O
untrained	O
mBERT	O
baseline	O
:	O
in	O
our	O
experiments	O
the	O
baseline	O
probes	O
for	O
both	O
English	O
and	O
German	O
attended	O
almost	O
exclusively	O
to	O
the	O
first	O
few	O
layers	O
of	O
the	O
encoder	O
,	O
independent	O
of	O
the	O
task	O
.	O
For	O
brevity	O
,	O
here	O
and	O
further	O
we	O
do	O
not	O
examine	O
baseline	O
mixing	O
weights	O
and	O
only	O
report	O
the	O
scores	O
.	O

Our	O
main	O
probing	O
results	O
mirror	O
the	O
findings	O
of	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
about	O
the	O
sequential	O
processing	O
order	O
in	O
BERT	O
.	O
We	O
observe	O
that	O
the	O
layer	O
utilization	O
among	O
tasks	O
(	O
Fig	O
.	O
2	O
)	O
generally	O
aligns	O
for	O
English	O
and	O
German	O
4	O
,	O
although	O
we	O
note	O
that	O
in	O
terms	O
of	O
center	O
-	O
of	O
-	O
gravity	O
mBERT	O
tends	O
to	O
utilize	O
deeper	O
layers	O
for	O
German	O
probes	O
.	O
Basic	O
word	O
-	O
level	O
tasks	O
are	O
indeed	O
processed	O
early	O
by	O
the	O
model	O
,	O
and	O
XNLI	O
probes	O
focus	O
on	O
deeper	O
levels	O
,	O
suggesting	O
that	O
the	O
representation	O
of	O
higher	O
-	O
level	O
semantic	O
phenomena	O
follows	O
the	O
encoding	O
of	O
syntax	O
and	O
predicate	O
semantics	O
.	O

The	O
Effect	O
of	O
Formalism	O

Using	O
separate	O
scalar	O
mixes	O
for	O
source	O
and	O
target	O
tokens	O
allows	O
us	O
to	O
explore	O
the	O
cross	O
-	O
formalism	O
encoding	O
of	O
role	O
semantics	O
by	O
mBERT	O
in	O
detail	O
.	O
For	O
both	O
English	O
and	O
German	O
role	O
labeling	O
,	O
the	O
probe	O
's	O
layer	O
utilization	O
drastically	O
differs	O
for	O
predicate	O
and	O
4	O
Echoing	O
the	O
recent	O
findings	O
on	O
mBERT	O
's	O
multilingual	O
capacity	O
(	O
Pires	O
et	O
al	O
.	O
,	O
2019;Kondratyuk	O
and	O
Straka	O
,	O
2019	O
[	O
6.16	O
]	O
xnli	O
[	O
6.28	O
]	O
en	O
Layer	O
[	O
4.61	O
]	O
[	O
5.2	O
]	O
[	O
5.09	O
]	O
[	O
5.75	O
]	O
[	O
6.01	O
]	O
[	O
5.99	O
]	O
[	O
5.18	O
]	O
[	O
5.24	O
]	O
[	O
5.13	O
]	O
[	O
6.12	O
]	O
[	O
6.06	O
]	O
[	O
5.75	O
]	O
[	O
6.15	O
]	O
de	O
argument	O
tokens	O
.	O
While	O
the	O
argument	O
representation	O
role	O
*	O
tgt	O
mostly	O
focuses	O
on	O
the	O
same	O
layers	O
as	O
the	O
dependency	O
parsing	O
probe	O
,	O
the	O
layer	O
utilization	O
of	O
the	O
predicates	O
role	O
*	O
src	O
is	O
affected	O
by	O
the	O
chosen	O
formalism	O
.	O
In	O
English	O
,	O
PropBank	O
predicate	O
token	O
mixing	O
weights	O
emphasize	O
the	O
same	O
layers	O
as	O
dependency	O
parsing	O
-in	O
line	O
with	O
the	O
previously	O
published	O
results	O
.	O
However	O
,	O
the	O
probes	O
for	O
VerbNet	O
and	O
FrameNet	O
predicates	O
(	O
role.vn	O
src	O
and	O
role.fn	O
src	O
)	O
utilize	O
the	O
layers	O
associated	O
with	O
ttype	O
and	O
lex.unit	O
that	O
contain	O
lexical	O
information	O
.	O
Coupled	O
with	O
the	O
fact	O
that	O
both	O
VerbNet	O
and	O
FrameNet	O
assign	O
semantic	O
roles	O
based	O
on	O
lexical	O
-	O
semantic	O
predicate	O
groupings	O
(	O
frames	O
in	O
FrameNet	O
and	O
verb	O
classes	O
in	O
VerbNet	O
)	O
,	O
this	O
suggests	O
that	O
the	O
lower	O
layers	O
of	O
mBERT	O
implicitly	O
encode	O
predicate	O
sense	O
information	O
;	O
moreover	O
,	O
sense	O
encoding	O
for	O
VerbNet	O
utilizes	O
deeper	O
layers	O
of	O
the	O
model	O
associated	O
with	O
syntax	O
,	O
in	O
line	O
with	O
Verb	O
-	O
Net	O
's	O
predicate	O
classification	O
strategy	O
.	O
This	O
finding	O
confirms	O
that	O
the	O
formalism	O
can	O
indeed	O
have	O
linguistically	O
meaningful	O
effects	O
on	O
probing	O
results	O
.	O

Anchor	O
Tasks	O
in	O
the	O
Pipeline	O

We	O
now	O
use	O
the	O
scalar	O
mixes	O
of	O
the	O
role	O
labeling	O
probes	O
as	O
target	O
tasks	O
,	O
and	O
lower	O
-	O
level	O
probes	O
as	O
anchor	O
tasks	O
to	O
qualitatively	O
explore	O
the	O
differences	O
between	O
how	O
our	O
role	O
probes	O
learn	O
to	O
represent	O
predicates	O
and	O
semantic	O
arguments	O
5	O
(	O
Fig	O
.	O
3	O
)	O
.	O
The	O
results	O
reveal	O
a	O
distinctive	O
pattern	O
that	O
confirms	O
our	O
previous	O
observations	O
:	O
while	O
Verb	O
-	O
Net	O
and	O
FrameNet	O
predicate	O
layer	O
utilization	O
src	O
is	O
similar	O
to	O
the	O
scalar	O
mixes	O
learned	O
for	O
ttype	O
and	O
lex.unit	O
,	O
the	O
learned	O
argument	O
representations	O
tgt	O
and	O
the	O
PropBank	O
predicate	O
attend	O
to	O
the	O
layers	O
associated	O
with	O
dependency	O
relation	O
and	O
POS	O
probes	O
.	O
Aside	O
from	O
the	O
PropBank	O
predicate	O
encoding	O
which	O
we	O
address	O
below	O
,	O
the	O
pattern	O
reproduces	O
for	O
English	O
and	O
German	O
.	O
This	O
aligns	O
with	O
the	O
traditional	O
separation	O
of	O
the	O
semantic	O
role	O
labeling	O
task	O
into	O
predicate	O
disambiguation	O
followed	O
by	O
semantic	O
argument	O
identification	O
and	O
labeling	O
,	O
along	O
with	O
the	O
feature	O
sets	O
employed	O
for	O
these	O
tasks	O
(	O
Björkelund	O
et	O
al	O
.	O
,	O
2009	O
)	O
.	O
Note	O
that	O
the	O
observation	O
about	O
the	O
pipeline	O
-	O
like	O
task	O
processing	O
within	O
the	O
BERT	O
encoders	O
thereby	O
holds	O
,	O
albeit	O
on	O
a	O
sub	O
-	O
task	O
level	O
.	O

Formalism	O
Implementations	O

Both	O
layer	O
and	O
anchor	O
task	O
analysis	O
reveal	O
a	O
prominent	O
discrepancy	O
between	O
English	O
and	O
German	O
role	O
probing	O
results	O
:	O
while	O
the	O
PropBank	O
predicate	O
layer	O
utilization	O
for	O
English	O
mostly	O
relies	O
on	O
syntactic	O
information	O
,	O
German	O
PropBank	O
predicates	O
behave	O
similarly	O
to	O
VerbNet	O
and	O
FrameNet	O
.	O
The	O
lack	O
of	O
systematic	O
cross	O
-	O
lingual	O
differences	O
between	O
layer	O
utilization	O
for	O
other	O
probing	O
tasks	O
6	O
allows	O
us	O
to	O
rule	O
out	O
the	O
effect	O
of	O
purely	O
typological	O
features	O
such	O
as	O
word	O
order	O
and	O
case	O
marking	O
as	O
a	O
likely	O
cause	O
.	O
The	O
difference	O
in	O
the	O
number	O
of	O
role	O
labels	O
for	O
English	O
and	O
German	O
PropBank	O
,	O
however	O
,	O
points	O
at	O
possible	O
qualitative	O
differences	O
in	O
the	O
labeling	O
schemes	O
(	O
Table	O
3	O
)	O
.	O
The	O
data	O
for	O
English	O
stems	O
from	O
the	O
token	O
-	O
level	O
alignment	O
in	O
SemLink	O
that	O
maps	O
the	O
original	O
PropBank	O
roles	O
to	O
Verb	O
-	O
Net	O
and	O
FrameNet	O
.	O
Role	O
annotations	O
for	O
German	O
have	O
a	O
different	O
lineage	O
:	O
they	O
originate	O
from	O
the	O
FrameNet	O
-	O
annotated	O
SALSA	O
corpus	O
(	O
Burchardt	O
et	O
al	O
.	O
,	O
2006	O
)	O
semi	O
-	O
automatically	O
converted	O
to	O
Prop	O
-	O
Bank	O
style	O
for	O
the	O
CoNLL-2009	O
shared	O
task	O
(	O
Hajič	O
et	O
al	O
.	O
,	O
2009	O
)	O
,	O
and	O
enriched	O
with	O
VerbNet	O
labels	O
in	O
SR3de	O
(	O
Mújdricza	O
-	O
Maydt	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
As	O
a	O
result	O
,	O
while	O
English	O
PropBank	O
labels	O
are	O
assigned	O
in	O
a	O
predicate	O
-	O
independent	O
manner	O
,	O
German	O
PropBank	O
,	O
following	O
the	O
same	O
numbered	O
labeling	O
scheme	O
,	O
keeps	O
this	O
scheme	O
consistent	O
within	O
the	O
frame	O
.	O
We	O
assume	O
that	O
this	O
incentivizes	O
the	O
probe	O
to	O
learn	O
semantic	O
verb	O
groupings	O
and	O
reflects	O
in	O
our	O
probing	O
results	O
.	O
The	O
ability	O
of	O
the	O
probe	O
to	O
detect	O
subtle	O
differences	O
between	O
formalism	O
implementations	O
constitutes	O
a	O
new	O
use	O
case	O
for	O
probing	O
,	O
and	O
a	O
promising	O
direction	O
for	O
future	O
studies	O
.	O

Encoding	O
of	O
Proto	O
-	O
Roles	O

We	O
now	O
turn	O
to	O
the	O
probing	O
results	O
for	O
decompositional	O
semantic	O
proto	O
-	O
role	O
labeling	O
tasks	O
.	O
Unlike	O
(	O
Tenney	O
et	O
al	O
.	O
,	O
2019b	O
)	O
who	O
used	O
a	O
multi	O
-	O
label	O
classification	O
probe	O
,	O
we	O
treat	O
SPR	O
properties	O
as	O
separate	O
regression	O
tasks	O
.	O
The	O
results	O
in	O
Table	O
6	O
show	O
that	O
the	O
performance	O
varies	O
by	O
property	O
,	O
with	O
some	O
of	O
the	O
properties	O
attaining	O
reasonably	O
high	O
R	O
2	O
scores	O
despite	O
the	O
simplicity	O
of	O
the	O
probe	O
architecture	O
and	O
the	O
small	O
dataset	O
size	O
.	O
We	O
observe	O
that	O
properties	O
associated	O
with	O
Proto	O
-	O
Agent	O
tend	O
to	O
perform	O
better	O
.	O
The	O
untrained	O
mBERT	O
baseline	O
performs	O
poorly	O
which	O
we	O
attribute	O
to	O
the	O
lack	O
of	O
data	O
and	O
the	O
finegrained	O
semantic	O
nature	O
of	O
the	O
task	O
.	O
Our	O
fine	O
-	O
grained	O
,	O
property	O
-	O
level	O
task	O
design	O
allows	O
for	O
more	O
detailed	O
insights	O
into	O
the	O
layer	O
utilization	O
by	O
the	O
SPR	O
probes	O
(	O
Fig	O
.	O
4	O
)	O
.	O
The	O
results	O
indicate	O
that	O
while	O
the	O
layer	O
utilization	O
on	O
the	O
predicate	O
side	O
(	O
src	O
)	O
shows	O
no	O
clear	O
preference	O
for	O
particular	O
layers	O
(	O
similar	O
to	O
the	O
results	O
obtained	O
by	O
Tenney	O
et	O
al	O
.	O
(	O
2019a	O
)	O
)	O
,	O
some	O
of	O
the	O
proto	O
-	O
role	O
features	O
follow	O
the	O
pattern	O
seen	O
in	O
the	O
categorical	O
role	O
labeling	O
and	O
dependency	O
parsing	O
tasks	O
for	O
the	O
argument	O
tokens	O
tgt	O
.	O
With	O
few	O
exceptions	O
,	O
we	O
observe	O
that	O
the	O
properties	O
displaying	O
that	O
behavior	O
are	O
Proto	O
-	O
Agent	O
properties	O
;	O
moreover	O
,	O
a	O
close	O
examination	O
of	O
the	O
results	O
on	O
syntactic	O
preference	O
by	O
Reisinger	O
et	O
al	O
.	O
(	O
2015	O
,	O
p.	O
483	O
)	O
reveals	O
that	O
these	O
properties	O
are	O
also	O
the	O
ones	O
with	O
strong	O
preference	O
for	O
the	O
subject	O
position	O
,	O
including	O
the	O
outlier	O
case	O
of	O
stationary	O
which	O
in	O
their	O
data	O
behaves	O
like	O
a	O
Proto	O
-	O
Agent	O
property	O
.	O
The	O
correspondence	O
is	O
not	O
strict	O
,	O
and	O
we	O
leave	O
an	O
in	O
-	O
depth	O
investigation	O
of	O
the	O
reasons	O
behind	O
these	O
discrepancies	O
for	O
follow	O
-	O
up	O
work	O
.	O

Conclusion	O

We	O
have	O
demonstrated	O
that	O
the	O
choice	O
of	O
linguistic	O
formalism	O
can	O
have	O
substantial	O
,	O
linguistically	O
meaningful	O
effects	O
on	O
role	O
-	O
semantic	O
probing	O
results	O
.	O
We	O
have	O
shown	O
how	O
probing	O
classifiers	O
can	O
be	O
used	O
to	O
detect	O
discrepancies	O
between	O
formalism	O
implementations	O
,	O
and	O
presented	O
evidence	O
of	O
semantic	O
proto	O
-	O
role	O
encoding	O
in	O
the	O
pre	O
-	O
trained	O
mBERT	O
model	O
.	O
Our	O
refined	O
implementation	O
of	O
the	O
edge	O
probing	O
framework	O
coupled	O
with	O
the	O
anchor	O
task	O
methodology	O
enabled	O
new	O
insights	O
into	O
the	O
processing	O
of	O
predicate	O
-	O
semantic	O
information	O
within	O
mBERT	O
.	O
Our	O
findings	O
suggest	O
that	O
linguistic	O
formalism	O
is	O
an	O
important	O
factor	O
to	O
be	O
accounted	O
for	O
in	O
probing	O
studies	O
.	O
This	O
prompts	O
several	O
recommendations	O
for	O
the	O
follow	O
-	O
up	O
probing	O
studies	O
.	O
First	O
,	O
the	O
formalism	O
and	O
implementation	O
used	O
to	O
prepare	O
the	O
linguistic	O
material	O
underlying	O
a	O
probing	O
study	O
should	O
be	O
always	O
explicitly	O
specified	O
.	O
Second	O
,	O
if	O
possible	O
,	O
results	O
on	O
multiple	O
formalisations	O
of	O
the	O
same	O
task	O
should	O
be	O
reported	O
and	O
validated	O
for	O
several	O
languages	O
.	O
Finally	O
,	O
assembling	O
corpora	O
with	O
parallel	O
cross	O
-	O
formalism	O
annotations	O
would	O
facilitate	O
further	O
research	O
on	O
the	O
effect	O
of	O
formalism	O
in	O
probing	O
.	O

While	O
our	O
work	O
illustrates	O
the	O
impact	O
of	O
formalism	O
using	O
a	O
single	O
task	O
and	O
a	O
single	O
probing	O
framework	O
,	O
the	O
influence	O
of	O
linguistic	O
formalism	O
per	O
se	O
is	O
likely	O
to	O
be	O
present	O
for	O
any	O
probing	O
setup	O
that	O
builds	O
upon	O
linguistic	O
material	O
.	O
An	O
investigation	O
of	O
how	O
,	O
whether	O
,	O
and	O
why	O
formalisms	O
and	O
their	O
implementations	O
affect	O
probing	O
results	O
for	O
tasks	O
beyond	O
role	O
labeling	O
and	O
for	O
frameworks	O
beyond	O
edge	O
probing	O
constitutes	O
an	O
exciting	O
avenue	O
for	O
future	O
research	O
.	O

Acknowledgments	O

This	O
work	O
has	O
been	O
funded	O
by	O
the	O
LOEWE	O
initiative	O
(	O
Hesse	O
,	O
Germany	O
)	O
within	O
the	O
emergenCITY	O
center	O
.	O

Pre	O
-	O
Training	O
Transformers	O
as	O
Energy	O
-	O
Based	O
Cloze	O
Models	O

We	O
introduce	O
Electric	O
,	O
an	O
energy	O
-	O
based	O
cloze	O
model	O
for	O
representation	O
learning	O
over	O
text	O
.	O
Like	O
BERT	O
,	O
it	O
is	O
a	O
conditional	O
generative	O
model	O
of	O
tokens	O
given	O
their	O
contexts	O
.	O
However	O
,	O
Electric	O
does	O
not	O
use	O
masking	O
or	O
output	O
a	O
full	O
distribution	O
over	O
tokens	O
that	O
could	O
occur	O
in	O
a	O
context	O
.	O
Instead	O
,	O
it	O
assigns	O
a	O
scalar	O
energy	O
score	O
to	O
each	O
input	O
token	O
indicating	O
how	O
likely	O
it	O
is	O
given	O
its	O
context	O
.	O
We	O
train	O
Electric	O
using	O
an	O
algorithm	O
based	O
on	O
noise	O
-	O
contrastive	O
estimation	O
and	O
elucidate	O
how	O
this	O
learning	O
objective	O
is	O
closely	O
related	O
to	O
the	O
recently	O
proposed	O
ELECTRA	O
pre	O
-	O
training	O
method	O
.	O
Electric	O
performs	O
well	O
when	O
transferred	O
to	O
downstream	O
tasks	O
and	O
is	O
particularly	O
effective	O
at	O
producing	O
likelihood	O
scores	O
for	O
text	O
:	O
it	O
reranks	O
speech	O
recognition	O
n	O
-	O
best	O
lists	O
better	O
than	O
language	O
models	O
and	O
much	O
faster	O
than	O
masked	O
language	O
models	O
.	O
Furthermore	O
,	O
it	O
offers	O
a	O
clearer	O
and	O
more	O
principled	O
view	O
of	O
what	O
ELECTRA	O
learns	O
during	O
pre	O
-	O
training	O
.	O

Introduction	O

The	O
cloze	O
task	O
(	O
Taylor	O
,	O
1953	O
)	O
of	O
predicting	O
the	O
identity	O
of	O
a	O
token	O
given	O
its	O
surrounding	O
context	O
has	O
proven	O
highly	O
effective	O
for	O
representation	O
learning	O
over	O
text	O
.	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
implements	O
the	O
cloze	O
task	O
by	O
replacing	O
input	O
tokens	O
with	O
[	O
MASK	O
]	O
,	O
but	O
this	O
approach	O
incurs	O
drawbacks	O
in	O
efficiency	O
(	O
only	O
15	O
%	O
of	O
tokens	O
are	O
masked	O
out	O
at	O
a	O
time	O
)	O
and	O
introduces	O
a	O
pre	O
-	O
train	O
/	O
fine	O
-	O
tune	O
mismatch	O
where	O
BERT	O
sees	O
[	O
MASK	O
]	O
tokens	O
in	O
training	O
but	O
not	O
in	O
fine	O
-	O
tuning	O
.	O
ELECTRA	O
(	O
Clark	O
et	O
al	O
.	O
,	O
2020	O
)	O
uses	O
a	O
different	O
pre	O
-	O
training	O
task	O
that	O
alleviates	O
these	O
disadvantages	O
.	O
Instead	O
of	O
masking	O
tokens	O
,	O
ELECTRA	O
replaces	O
some	O
input	O
tokens	O
with	O
fakes	O
sampled	O
from	O
a	O
small	O
generator	O
network	O
.	O
The	O
pre	O
-	O
training	O
task	O
is	O
then	O
to	O
distinguish	O
the	O
original	O
vs.	O
replaced	O
tokens	O
.	O
While	O
on	O
the	O
surface	O
it	O
appears	O
quite	O
different	O
from	O
BERT	O
,	O
in	O
this	O
paper	O
we	O
elucidate	O
a	O
close	O
connection	O
between	O
ELECTRA	O
and	O
cloze	O
modeling	O
.	O
In	O
particular	O
,	O
we	O
develop	O
a	O
new	O
way	O
of	O
implementing	O
the	O
cloze	O
task	O
using	O
an	O
energy	O
-	O
based	O
model	O
(	O
EBM	O
)	O
.	O
Then	O
we	O
show	O
the	O
resulting	O
model	O
,	O
which	O
we	O
call	O
Electric	O
,	O
is	O
closely	O
related	O
to	O
ELECTRA	O
,	O
as	O
well	O
as	O
being	O
useful	O
in	O
its	O
own	O
right	O
for	O
some	O
applications	O
.	O
1	O
EBMs	O
learn	O
an	O
energy	O
function	O
that	O
assigns	O
low	O
energy	O
values	O
to	O
inputs	O
in	O
the	O
data	O
distribution	O
and	O
high	O
energy	O
values	O
to	O
other	O
inputs	O
.	O
They	O
are	O
flexible	O
because	O
they	O
do	O
not	O
have	O
to	O
compute	O
normalized	O
probabilities	O
.	O
For	O
example	O
,	O
Electric	O
does	O
not	O
use	O
masking	O
or	O
an	O
output	O
softmax	O
,	O
instead	O
producing	O
a	O
scalar	O
energy	O
score	O
for	O
each	O
token	O
where	O
a	O
low	O
energy	O
indicates	O
the	O
token	O
is	O
likely	O
given	O
its	O
context	O
.	O
Unlike	O
with	O
BERT	O
,	O
these	O
likelihood	O
scores	O
can	O
be	O
computed	O
simultaneously	O
for	O
all	O
input	O
tokens	O
rather	O
than	O
only	O
for	O
a	O
small	O
masked	O
-	O
out	O
subset	O
.	O
We	O
propose	O
a	O
training	O
algorithm	O
for	O
Electric	O
that	O
efficiently	O
approximates	O
a	O
loss	O
based	O
on	O
noise	O
-	O
contrastive	O
estimation	O
(	O
Gutmann	O
and	O
Hyvärinen	O
,	O
2010	O
)	O
.	O
Then	O
we	O
show	O
that	O
this	O
training	O
algorithm	O
is	O
closely	O
related	O
to	O
ELECTRA	O
;	O
in	O
fact	O
,	O
ELECTRA	O
can	O
be	O
viewed	O
as	O
a	O
variant	O
of	O
Electric	O
using	O
negative	O
sampling	O
instead	O
of	O
noise	O
-	O
contrastive	O
estimation	O
.	O

We	O
evaluate	O
Electric	O
on	O
GLUE	O
and	O
SQuAD	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
where	O
Electric	O
substantially	O
outperforms	O
BERT	O
but	O
slightly	O
under	O
-	O
performs	O
ELECTRA	O
.	O
However	O
,	O
Electric	O
is	O
particularly	O
useful	O
in	O
its	O
ability	O
to	O
efficiently	O
produce	O
pseudo	O
-	O
likelihood	O
scores	O
(	O
Salazar	O
et	O
al	O
.	O
,	O
2020	O
)	O
for	O
text	O
:	O
Electric	O
is	O
better	O
at	O
re	O
-	O
ranking	O
the	O
outputs	O
of	O
a	O
speech	O
recognition	O
system	O
than	O
GPT-2	O
(	O
Radford	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
is	O
much	O
faster	O
at	O
re	O
-	O
ranking	O
than	O
BERT	O
because	O
it	O
scores	O
all	O
input	O
tokens	O
simultaneously	O
rather	O
than	O
having	O
to	O
be	O
run	O
multiple	O
times	O
with	O
different	O
tokens	O
masked	O
out	O
.	O
In	O
total	O
,	O
investigating	O
Electric	O
leads	O
to	O
a	O
more	O
principled	O
understanding	O
of	O
ELECTRA	O
and	O
our	O
results	O
suggest	O
that	O
EBMs	O
are	O
a	O
promising	O
alternative	O
to	O
the	O
standard	O
generative	O
models	O
currently	O
used	O
for	O
language	O
representation	O
learning	O
.	O

Method	O

BERT	O
and	O
related	O
pre	O
-	O
training	O
methods	O
(	O
Baevski	O
et	O
al	O
.	O
,	O
2019;Lan	O
et	O
al	O
.	O
,	O
2020	O
)	O
train	O
a	O
large	O
neural	O
network	O
to	O
perform	O
the	O
cloze	O
task	O
.	O
These	O
models	O
learn	O
the	O
probability	O
p	O
data	O
(	O
x	O
t	O
|x	O
\t	O
)	O
of	O
a	O
token	O
x	O
t	O
occurring	O
in	O
the	O
surrounding	O
context	O

x	O
\t	O
=	O
[	O
x	O
1	O
,	O
...	O
,	O
x	O
t−1	O
,	O
x	O
t+1	O
,	O
...	O
,	O
x	O
n	O
]	O
.	O

Typically	O
the	O
context	O
is	O
represented	O
as	O
the	O
input	O
sequence	O
with	O
x	O
t	O
replaced	O
by	O
a	O
special	O
[	O
MASK]placeholder	O
token	O
.	O
This	O
masked	O
sequence	O
is	O
encoded	O
into	O
vector	O
representations	O
by	O
a	O
transformer	O
network	O
(	O
Vaswani	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
Then	O
the	O
representation	O
at	O
position	O
t	O
is	O
passed	O
into	O
a	O
softmax	O
layer	O
to	O
produce	O
a	O
distribution	O
over	O
tokens	O
p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
for	O
the	O
position	O
.	O

The	O
Electric	O
Model	O

Electric	O
also	O
models	O
p	O
data	O
(	O
x	O
t	O
|x	O
\t	O
)	O
,	O
but	O
does	O
not	O
use	O
masking	O
or	O
a	O
softmax	O
layer	O
.	O
Electric	O
first	O
maps	O
the	O
unmasked	O
input	O
x	O
=	O
[	O
x	O
1	O
,	O
...	O
,	O
x	O
n	O
]	O
into	O
contextualized	O
vector	O
representations	O
h(x	O
)	O
=	O
[	O
h	O
1	O
,	O
...	O
,	O
h	O
n	O
]	O
using	O
a	O
transformer	O
network	O
.	O
The	O
model	O
assigns	O
a	O
given	O
position	O
t	O
an	O
energy	O
score	O
E(x	O
)	O
t	O
=	O
w	O
T	O
h(x	O
)	O
t	O
using	O
a	O
learned	O
weight	O
vector	O
w.	O
The	O
energy	O
function	O
defines	O
a	O
distribution	O
over	O
the	O
possible	O
tokens	O
at	O
position	O
t	O
as	O

p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
=	O
exp	O
(	O
−E(x	O
)	O
t	O
)	O
/Z(x	O
\t	O
)	O
=	O
exp	O
(	O
−E(x	O
)	O
t	O
)	O
x	O
∈V	O
exp	O
(	O
−E(REPLACE(x	O
,	O
t	O
,	O
x	O
)	O
)	O
t	O
)	O

where	O
REPLACE(x	O
,	O
t	O
,	O
x	O
)	O
denotes	O
replacing	O
the	O
token	O
at	O
position	O
t	O
with	O
x	O
and	O
V	O
is	O
the	O
vocabulary	O
,	O
in	O
practice	O
usually	O
word	O
pieces	O
(	O
Sennrich	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
Unlike	O
with	O
BERT	O
,	O
which	O
produces	O
the	O
probabilities	O
for	O
all	O
possible	O
tokens	O
x	O
using	O
a	O
softmax	O
layer	O
,	O
a	O
candidate	O
x	O
is	O
passed	O
in	O
as	O
input	O
to	O
the	O
transformer	O
.	O
As	O
a	O
result	O
,	O
computing	O
p	O
θ	O
is	O
prohibitively	O
expensive	O
because	O
the	O
partition	O
function	O
Z	O
θ	O
(	O
x	O
\t	O
)	O
requires	O
running	O
the	O
transformer	O
|V|	O
times	O
;	O
unlike	O
most	O
EBMs	O
,	O
the	O
intractability	O
of	O
Z	O
θ	O
(	O
x	O
\t	O
)	O
is	O
due	O
to	O
the	O
expensive	O
scoring	O
function	O
rather	O
than	O
having	O
a	O
large	O
sample	O
space	O
.	O

As	O
computing	O
the	O
exact	O
likelihood	O
is	O
intractable	O
,	O
training	O
energy	O
-	O
based	O
models	O
such	O
as	O
Electric	O
with	O
standard	O
maximum	O
-	O
likelihood	O
estimation	O
is	O
not	O
possible	O
.	O
Instead	O
,	O
we	O
use	O
(	O
conditional	O
)	O
Noise	O
-	O
Contrastive	O
Estimation	O
(	O
NCE	O
)	O
(	O
Gutmann	O
and	O
Hyvärinen	O
,	O
2010;Ma	O
and	O
Collins	O
,	O
2018	O
)	O
,	O
which	O
provides	O
a	O
way	O
of	O
efficiently	O
training	O
an	O
unnormalized	O
model	O
that	O
does	O
not	O
compute	O
Z	O
θ	O
(	O
x	O
\t	O
)	O
.	O
NCE	O
learns	O
the	O
parameters	O
of	O
a	O
model	O
by	O
defining	O
a	O
binary	O
classification	O
task	O
where	O
samples	O
from	O
the	O
data	O
distribution	O
have	O
to	O
be	O
distinguished	O
from	O
samples	O
generated	O
by	O
a	O
noise	O
distribution	O
q(x	O
t	O
|x	O
\t	O
)	O
.	O
First	O
,	O
we	O
define	O
the	O
un	O
-	O
normalized	O
output	O
p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
=	O
exp	O
(	O
−E(x	O
)	O
t	O
)	O

Operationally	O
,	O
NCE	O
can	O
be	O
viewed	O
as	O
follows	O
:	O

•	O
A	O
positive	O
data	O
point	O
is	O
a	O
text	O
sequence	O
x	O
from	O
the	O
data	O
and	O
position	O
in	O
the	O
sequence	O
t.	O

•	O
A	O
negative	O
data	O
point	O
is	O
the	O
same	O
except	O
x	O
t	O
,	O
the	O
token	O
at	O
position	O
t	O
,	O
is	O
replaced	O
with	O
a	O
noise	O
tokenx	O
t	O
sampled	O
from	O
q.	O

•	O
Define	O
a	O
binary	O
classifier	O
D	O
that	O
estimates	O
the	O
probability	O
of	O
a	O
data	O
point	O
being	O
positive	O
as	O

n	O
•p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
n	O
•p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
+	O
k	O
•	O
q(x	O
t	O
|x	O
\t	O
)	O

•	O
The	O
binary	O
classifier	O
is	O
trained	O
to	O
distinguish	O
positive	O
vs	O
negative	O
data	O
points	O
,	O
with	O
k	O
negatives	O
sampled	O
for	O
every	O
n	O
positive	O
data	O
points	O
.	O

Formally	O
,	O
the	O
NCE	O
loss	O
L(θ	O
)	O
is	O

n	O
•	O
E	O
x	O
,	O
t	O
−	O
log	O
n	O
•p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
n	O
•p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
+	O
k	O
•	O
q(x	O
t	O
|x	O
\t	O
)	O
+	O
k	O
•	O
E	O
x	O
,	O
t	O
xt∼q	O
−	O
log	O
k	O
•	O
q(x	O
t	O
|x	O
\t	O
)	O
n	O
•p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
+	O
k	O
•	O
q(x	O
t	O
|x	O
\t	O
)	O

This	O
loss	O
is	O
minimized	O
whenp	O
θ	O
matches	O
the	O
data	O
distribution	O
p	O
data	O
(	O
Gutmann	O
and	O
Hyvärinen	O
,	O
2010	O
)	O
.	O
A	O
consequence	O
of	O
this	O
property	O
is	O
that	O
the	O
model	O
learns	O
to	O
be	O
self	O
-	O
normalized	O
such	O
that	O
Z	O
θ	O
(	O
x	O
\t	O
)	O
=	O
1	O
.	O

Training	O
Algorithm	O

To	O
minimize	O
the	O
loss	O
,	O
the	O
expectations	O
could	O
be	O
approximated	O
by	O
sampling	O
as	O
shown	O
in	O
Algorithm	O
1	O
.	O
Taking	O
the	O
gradient	O
of	O
this	O
estimated	O
loss	O
produces	O
Algorithm	O
1	O
Naive	O
NCE	O
loss	O
estimation	O
Given	O
:	O
Input	O
sequence	O
x	O
,	O
number	O
of	O
negative	O
samples	O
k	O
,	O
noise	O
distribution	O
q	O
,	O
modelp	O
θ	O
.	O

1	O
.	O
Initialize	O
the	O
loss	O
as	O

n	O
t=1	O
−	O
log	O
n•p	O
θ	O
(	O
xt|x	O
\t	O
)	O
n•p	O
θ	O
(	O
xt|x	O
\t	O
)	O
+	O
k•q(xt|x	O
\t	O
)	O
.	O
2	O
.	O
Sample	O
k	O
negative	O
samples	O
according	O
to	O
t	O
∼	O
unif{1	O
,	O
n},x	O
t	O
∼	O
q(x	O
t	O
|x	O
\t	O
)	O
.	O
3	O
.	O
For	O
each	O
negative	O
sample	O
,	O
add	O
to	O
the	O
loss	O
−	O
log	O
k•q(xt|x	O
\t	O
)	O
n•p	O
θ	O
(	O
xt|x	O
\t	O
)	O
+	O
k•q(xt|x	O
\t	O
)	O
.	O

an	O
unbiased	O
estimate	O
of	O
∇	O
θ	O
L(θ	O
)	O
.	O
However	O
,	O
this	O
algorithm	O
is	O
computationally	O
expensive	O
to	O
run	O
,	O
since	O
it	O
requires	O
k	O
+	O
1	O
forward	O
passes	O
through	O
the	O
transformer	O
to	O
compute	O
thep	O
θ	O
s	O
(	O
once	O
for	O
the	O
positive	O
samples	O
and	O
once	O
for	O
each	O
negative	O
sample	O
)	O
.	O
We	O
propose	O
a	O
much	O
more	O
efficient	O
approach	O
that	O
replaces	O
k	O
input	O
tokens	O
with	O
noise	O
samples	O
simultaneously	O
shown	O
in	O
Algorithm	O
2	O
.	O
It	O
requires	O
just	O

Algorithm	O
2	O
Efficient	O
NCE	O
loss	O
estimation	O

Given	O
:	O
Input	O
sequence	O
x	O
,	O
number	O
of	O
negative	O
samples	O
k	O
,	O
noise	O
distribution	O
q	O
,	O
modelp	O
θ	O
.	O
1	O
.	O
Pick	O
k	O
unique	O
random	O
positions	O
R	O
=	O
{	O
r	O
1	O
,	O
...	O
,	O
r	O
k	O
}	O
where	O
each	O
r	O
i	O
is	O
1	O
≤	O
r	O
i	O
≤	O
n.	O

2	O
.	O
Replace	O
the	O
k	O
random	O
positions	O
with	O
negative	O
samples	O
:	O

x	O
i	O
∼	O
q(x	O
i	O
|x	O
\i	O
)	O
for	O
i	O
∈	O
R	O
,	O
x	O
noised	O
=	O
REPLACE(x	O
,	O
R	O
,	O
X	O
)	O
.	O
3	O
.	O

For	O
each	O
position	O
t	O
=	O
1	O
to	O
n	O
:	O
add	O
to	O
the	O
loss	O
−	O
log	O

k•q(xt|x	O
\t	O
)	O
(	O
n−k)•p	O
θ	O
(	O
xt|x	O
noised	O
\t	O
)	O
+	O
k•q(xt|x	O
\t	O
)	O
if	O
t	O
∈	O
R	O
−	O
log	O
(	O
n−k)•p	O
θ	O
(	O
xt|x	O
noised	O
\t	O
)	O
(	O
n−k)•p	O
θ	O
(	O
xt|x	O
noised	O
\t	O
)	O
+	O
k•q(xt|x	O
\t	O
)	O
otherwise	O

one	O
pass	O
through	O
the	O
transformer	O
for	O
k	O
noise	O
samples	O
and	O
n	O
−	O
k	O
data	O
samples	O
.	O
However	O
,	O
this	O
procedure	O
only	O
truly	O
minimizes	O
L	O
ifp	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
=	O
p	O
θ	O
(	O
x	O
t	O
|x	O
noised	O
\t	O
)	O
.	O
To	O
apply	O
this	O
efficiency	O
trick	O
we	O
are	O
making	O
the	O
assumption	O
they	O
are	O
approximately	O
equal	O
,	O
which	O
we	O
argue	O
is	O
reasonable	O
because	O
(	O
1	O
)	O
we	O
choose	O
a	O
small	O
k	O
of	O
0.15n	O
and	O
(	O
2	O
)	O
q	O
is	O
trained	O
to	O
be	O
close	O
to	O
the	O
data	O
distribution	O
(	O
see	O
below	O
)	O
.	O
This	O
efficiency	O
trick	O
is	O
analogous	O
to	O
BERT	O
masking	O
out	O
multiple	O
tokens	O
per	O
input	O
sequence	O
.	O

Noise	O
Distribution	O

The	O
noise	O
distribution	O
q	O
comes	O
from	O
a	O
neural	O
network	O
trained	O
to	O
match	O
p	O
data	O
.	O
NCE	O
commonly	O
employs	O
this	O
idea	O
to	O
ensure	O
the	O
classification	O
task	O
is	O
sufficiently	O
challenging	O
for	O
the	O
model	O
(	O
Gutmann	O
and	O
Hyvärinen	O
,	O
2010;Wang	O
and	O
Ou	O
,	O
2018	O
)	O
.	O
In	O
particular	O
,	O
we	O
use	O
a	O
two	O
-	O
tower	O
cloze	O
model	O
as	O
proposed	O
by	O
Baevski	O
et	O
al	O
.	O
(	O
2019	O
)	O
,	O
which	O
is	O
more	O
accurate	O
than	O
a	O
language	O
model	O
because	O
it	O
uses	O
context	O
to	O
both	O
sides	O
of	O
each	O
token	O
.	O
The	O
model	O
runs	O
two	O
transformers	O
T	O
LTR	O
and	O
T	O
RTL	O
over	O
the	O
input	O
sequence	O
.	O
These	O
transformers	O
apply	O
causal	O
masking	O
so	O
one	O
processes	O
the	O
sequence	O
left	O
-	O
to	O
-	O
right	O
and	O
the	O
other	O
operates	O
right	O
-	O
to	O
-	O
left	O
.	O
The	O
model	O
's	O
predictions	O
come	O
from	O
a	O
softmax	O
layer	O
applied	O
to	O
the	O
concatenated	O
states	O
of	O
the	O
two	O
transformers	O
:	O

−	O
→	O
h	O
=	O
T	O
LTR	O
(	O
x	O
)	O
,	O
←	O
−	O
h	O
=	O
T	O
RTL	O
(	O
x	O
)	O
q(x	O
t	O
|x	O
\t	O
)	O
=	O
softmax(W	O
[	O
−	O
→	O
h	O
t−1	O
,	O
←	O
−	O
h	O
t+1	O
]	O
)	O
xt	O

The	O
noise	O
distribution	O
is	O
trained	O
simultaneously	O
with	O
Electric	O
using	O
standard	O
maximum	O
likelihood	O
estimation	O
over	O
the	O
data	O
.	O
The	O
model	O
producing	O
the	O
noise	O
distribution	O
is	O
much	O
smaller	O
than	O
Electric	O
to	O
reduce	O
the	O
computational	O
overhead	O
.	O

Connection	O
to	O
ELECTRA	O

Electric	O
is	O
closely	O
related	O
to	O
the	O
ELECTRA	O
pretraining	O
method	O
.	O
ELECTRA	O
also	O
trains	O
a	O
binary	O
classifier	O
(	O
the	O
"	O
discriminator	O
"	O
)	O
to	O
distinguish	O
data	O
tokens	O
from	O
noise	O
tokens	O
produced	O
by	O
a	O
"	O
generator	O
"	O
network	O
.	O
However	O
,	O
ELECTRA	O
's	O
classifier	O
is	O
simply	O
a	O
sigmoid	O
layer	O
on	O
top	O
of	O
the	O
transformer	O
:	O
it	O
models	O
the	O
probability	O
of	O
a	O
token	O
being	O
negative	O
(	O
i.e.	O
,	O
as	O
replaced	O
by	O
a	O
noise	O
sample	O
)	O
as	O
σ(E(x	O
)	O
t	O
)	O
where	O
σ	O
denotes	O
the	O
sigmoid	O
function	O
.	O
Electric	O
on	O
the	O
other	O
hand	O
models	O
this	O
probability	O
as	O

k	O
•	O
q(x|x	O
\t	O
)	O
n	O
•	O
exp	O
(	O
−E(x	O
)	O
t	O
)	O
+	O
k	O
•	O
q(x|x	O
\t	O
)	O
=	O
σ	O
E(x	O
)	O
t	O
+	O
log	O
k	O
•	O
q(x|x	O
\t	O
)	O
n	O

While	O
ELECTRA	O
learns	O
whether	O
a	O
token	O
is	O
more	O
likely	O
to	O
come	O
from	O
the	O
data	O
distribution	O
p	O
data	O
or	O
noise	O
distribution	O
q	O
,	O
Electric	O
only	O
learns	O
p	O
data	O
because	O
q	O
is	O
passed	O
into	O
the	O
model	O
directly	O
.	O
This	O
difference	O
is	O
analogous	O
to	O
using	O
negative	O
sampling	O
(	O
Mikolov	O
et	O
al	O
.	O
,	O
2013	O
)	O
vs.	O
noise	O
-	O
contrastive	O
estimation	O
(	O
Mnih	O
and	O
Kavukcuoglu	O
,	O
2013	O
)	O
for	O
learning	O
word	O
embeddings	O
.	O
A	O
disadvantage	O
of	O
Electric	O
compared	O
to	O
ELEC	O
-	O
TRA	O
is	O
that	O
it	O
is	O
less	O
flexible	O
in	O
the	O
choice	O
of	O
noise	O
distribution	O
.	O
Since	O
ELECTRA	O
's	O
binary	O
classifier	O
does	O
not	O
need	O
to	O
access	O
q	O
,	O
its	O
q	O
only	O
needs	O
to	O
be	O
defined	O
for	O
negative	O
sample	O
positions	O
in	O
the	O
input	O
sequence	O
.	O
Therefore	O
ELECTRA	O
can	O
use	O
a	O
masked	O
language	O
model	O
rather	O
than	O
a	O
two	O
-	O
tower	O
cloze	O
model	O
for	O
q.	O
An	O
advantage	O
of	O
Electric	O
is	O
that	O
it	O
directly	O
provides	O
(	O
un	O
-	O
normalized	O
)	O
probabilitieŝ	O
p	O
θ	O
for	O
tokens	O
,	O
making	O
it	O
useful	O
for	O
applications	O
such	O
as	O
re	O
-	O
ranking	O
the	O
outputs	O
of	O
text	O
generation	O
systems	O
.	O
The	O
differences	O
between	O
ELECTRA	O
and	O
Electric	O
are	O
summarized	O
below	O
:	O

Model	O
Noise	O
Dist	O
.	O
Binary	O
Classifier	O
Electric	O
Two	O
-	O
Tower	O
Cloze	O
Model	O
σ	O
E(x)t	O
+	O
log	O
k•q(x|x	O
\t	O
)	O
n	O
ELECTRA	O
Masked	O
LM	O
σ(E(x)t	O
)	O

Experiments	O

We	O
train	O
two	O
Electric	O
models	O
the	O
same	O
size	O
as	O
BERT	O
-	O
Base	O
(	O
110	O
M	O
parameters	O
):	O
one	O
on	O
Wikipedia	O
and	O
BooksCorpus	O
(	O
Zhu	O
et	O
al	O
.	O
,	O
2015	O
)	O
for	O
comparison	O
with	O
BERT	O
and	O
one	O
on	O
OpenWebTextCorpus	O
(	O
Gokaslan	O
and	O
Cohen	O
,	O
2019	O
)	O
for	O
comparison	O
2	O
with	O
GPT-2	O
.	O
The	O
noise	O
distribution	O
transformers	O
T	O
LTR	O
and	O
T	O
RTL	O
are	O
1/4	O
the	O
hidden	O
size	O
of	O
Electric	O
.	O
We	O
do	O
no	O
hyperparameter	O
tuning	O
,	O
using	O
the	O
same	O
hyperparameter	O
values	O
as	O
ELECTRA	O
.	O
Further	O
details	O
on	O
training	O
are	O
in	O
the	O
appendix	O
.	O

Transfer	O
to	O
Downstream	O
Tasks	O

We	O
evaluate	O
fine	O
-	O
tuning	O
the	O
Electric	O
model	O
on	O
the	O
GLUE	O
natural	O
language	O
understanding	O
benchmark	O
and	O
the	O
SQuAD	O
2.0	O
question	O
answering	O
dataset	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
We	O
report	O
exact	O
-	O
match	O
for	O
SQuAD	O
,	O
average	O
score	O
3	O
over	O
the	O
GLUE	O
tasks	O
4	O
,	O
and	O
accuracy	O
on	O
the	O
multi	O
-	O
genre	O
natural	O
language	O
inference	O
GLUE	O
task	O
.	O
Reported	O
scores	O
are	O
medians	O
over	O
10	O
fine	O
-	O
tuning	O
runs	O
with	O
different	O
random	O
seeds	O
.	O
We	O
use	O
the	O
same	O
finetuning	O
setup	O
and	O
hyperparameters	O
as	O
ELECTRA	O
.	O

Results	O
are	O
shown	O
in	O
Table	O
1	O
.	O
Electric	O
scores	O
better	O
than	O
BERT	O
,	O
showing	O
the	O
energy	O
-	O
based	O
formulation	O
improves	O
cloze	O
model	O
pre	O
-	O
training	O
.	O
However	O
,	O
Electric	O
scores	O
slightly	O
lower	O
than	O
ELECTRA	O
.	O
One	O
possible	O
explanation	O
is	O
that	O
Electric	O
's	O
noise	O
distribution	O
is	O
worse	O
because	O
a	O
two	O
-	O
tower	O
cloze	O
model	O
is	O
less	O
expressive	O
than	O
a	O
masked	O
LM	O
.	O
We	O
tested	O
this	O
hypothesis	O
by	O
training	O
ELECTRA	O
with	O
the	O
same	O
two	O
-	O
tower	O
noise	O
model	O
as	O
Electric	O
.	O
Performance	O
did	O
indeed	O
go	O
down	O
,	O
but	O
it	O
only	O
explained	O
about	O
half	O
the	O
gap	O
.	O
The	O
surprising	O
drop	O
in	O
performance	O
suggests	O
that	O
learning	O
the	O
difference	O
between	O
the	O
data	O
and	O
generations	O
from	O
a	O
low	O
-	O
capacity	O
model	O
leads	O
to	O
better	O
representations	O
than	O
only	O
learning	O
the	O
data	O
distribution	O
,	O
but	O
we	O
believe	O
further	O
research	O
is	O
needed	O
to	O
fully	O
understand	O
the	O
discrepancy	O
.	O

Fast	O
Pseudo	O
-	O
Log	O
-	O
Likelihood	O
Scoring	O

An	O
advantage	O
of	O
Electric	O
over	O
BERT	O
is	O
that	O
it	O
can	O
efficiently	O
produce	O
pseudo	O
-	O
log	O
-	O
likelihood	O
(	O
PLL	O
)	O
scores	O
for	O
text	O
(	O
Wang	O
and	O
Cho	O
,	O
2019	O
)	O
.	O
PLLs	O
for	O
Electric	O
are	O

PLL(x	O
)	O
=	O
n	O
t=1	O
log(p	O
θ	O
(	O
x	O
t	O
|x	O
\t	O
)	O
)	O
=	O
n	O
t=1	O
−E(x	O
)	O
t	O

PLLs	O
can	O
be	O
used	O
to	O
re	O
-	O
rank	O
the	O
outputs	O
of	O
an	O
NMT	O
or	O
ASR	O
system	O
.	O
While	O
historically	O
log	O
-	O
likelihoods	O
from	O
language	O
models	O
have	O
been	O
used	O
for	O
such	O
reranking	O
,	O
recent	O
work	O
has	O
demonstrated	O
that	O
PLLs	O
from	O
masked	O
language	O
models	O
perform	O
better	O
(	O
Shin	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
However	O
,	O
computing	O
PLLs	O
from	O
a	O
masked	O
language	O
model	O
requires	O
n	O
passes	O
of	O
the	O
transformer	O
:	O
once	O
with	O
each	O
token	O
masked	O
out	O
.	O
Salazar	O
et	O
al	O
.	O
(	O
2020	O
)	O
suggest	O
distilling	O
BERT	O
into	O
a	O
model	O
that	O
uses	O
no	O
masking	O
to	O
avoid	O
this	O
cost	O
,	O
but	O
this	O
model	O
considerably	O
under	O
-	O
performed	O
regular	O
LMs	O
in	O
their	O
experiments	O
.	O

Electric	O
can	O
produce	O
PLLs	O
for	O
all	O
input	O
tokens	O
in	O
a	O
single	O
pass	O
like	O
a	O
LM	O
while	O
being	O
bidirectional	O
like	O
a	O
masked	O
LM	O
.	O
We	O
use	O
the	O
PLLs	O
from	O
Electric	O
for	O
re	O
-	O
ranking	O
the	O
100	O
-	O
best	O
hypotheses	O
of	O
a	O
5	O
-	O
layer	O
BLSTMP	O
model	O
from	O
ESPnet	O
(	O
Watanabe	O
et	O
al	O
.	O
,	O
2018	O
)	O
on	O
the	O
960	O
-	O
hour	O
LibriSpeech	O
corpus	O
(	O
Panayotov	O
et	O
al	O
.	O
,	O
2015	O
)	O
following	O
the	O
same	O
experimental	O
setup	O
and	O
using	O
the	O
same	O
n	O
-	O
best	O
lists	O
as	O
Salazar	O
et	O
al	O
.	O
(	O
2020	O
)	O
.	O
Given	O
speech	O
features	O
s	O
and	O
speech	O
recognition	O
model	O
f	O
the	O
re	O
-	O
ranked	O
output	O
is	O
arg	O
max	O

x∈n	O
-	O
best(f	O
,	O
s	O
)	O
f	O
(	O
x|s	O
)	O
+	O
λPLL(x	O
)	O

where	O
n	O
-	O
best(f	O
,	O
s	O
)	O
consists	O
of	O
the	O
top	O
n	O
(	O
we	O
use	O
n	O
=	O
100	O
)	O
predictions	O
from	O
the	O
speech	O
recognition	O
model	O
found	O
with	O
beam	O
search	O
,	O
f	O
(	O
x|s	O
)	O
is	O
the	O
score	O
the	O
speech	O
model	O
assigns	O
the	O
candidate	O
output	O
sequence	O
x.	O
We	O
select	O
the	O
best	O
λ	O
on	O
the	O
dev	O
set	O
out	O
of	O
[	O
0.05	O
,	O
0.1	O
,	O
...	O
,	O
0.95	O
,	O
1.0	O
]	O
,	O
with	O
different	O
λs	O
selected	O
for	O
the	O
"	O
clean	O
"	O
and	O
"	O
other	O
"	O
portions	O
of	O
the	O
data	O
.	O

We	O
compare	O
Electric	O
against	O
GPT-2	O
(	O
Radford	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
and	O
two	O
baseline	O
systems	O
that	O
are	O
bidirectional	O
while	O
only	O
requiring	O
a	O
single	O
transformer	O
pass	O
like	O
Electric	O
.	O
TwoTower	O
is	O
a	O
two	O
-	O
tower	O
cloze	O
model	O
similar	O
to	O
Electric	O
's	O
noise	O
distribution	O
,	O
but	O
of	O
the	O
same	O
size	O
as	O
Electric	O
.	O
ELECTRA	O
-	O
TT	O
is	O
identical	O
to	O
ELECTRA	O
except	O
it	O
uses	O
a	O
two	O
-	O
tower	O
noise	O
distribution	O
rather	O
than	O
a	O
masked	O
language	O
model	O
.	O
5	O
The	O
noise	O
distribution	O
probabilities	O
and	O
binary	O
classifiers	O
scores	O
of	O
ELECTRA	O
can	O
be	O
combined	O
to	O
assign	O
probabilities	O
for	O
tokens	O
as	O
shown	O
in	O
Appendix	O
G	O
of	O
the	O
ELECTRA	O
paper	O
.	O

Results	O
are	O
shown	O
in	O
Table	O
2	O
.	O
Electric	O
scores	O
better	O
than	O
GPT-2	O
when	O
trained	O
on	O
comparable	O
data	O
.	O
While	O
scoring	O
worse	O
than	O
BERT	O
,	O
Electric	O
is	O
much	O
faster	O
to	O
run	O
.	O
It	O
also	O
slightly	O
outperforms	O
ELECTRA	O
-	O
TT	O
,	O
which	O
is	O
consistent	O
with	O
the	O
finding	O
from	O
Labeau	O
and	O
Allauzen	O
(	O
2018	O
)	O
that	O
NCE	O
outperforms	O
negative	O
sampling	O
for	O
training	O
language	O
models	O
.	O
Furthermore	O
,	O
Electric	O
is	O
simpler	O
and	O
faster	O
than	O
ELETRA	O
-	O
TT	O
in	O
that	O
it	O
does	O
not	O
require	O
running	O
the	O
generator	O
to	O
produce	O
PLL	O
scores	O
.	O
TwoTower	O
scores	O
lower	O
than	O
Electric	O
,	O
presumably	O
because	O
it	O
is	O
not	O
a	O
"	O
deeply	O
"	O
bidirectional	O
model	O
and	O
instead	O
just	O
concatenates	O
forward	O
and	O
backward	O
hidden	O
states	O
.	O

Related	O
Work	O

Language	O
modeling	O
(	O
Dai	O
and	O
Le	O
,	O
2015;Radford	O
et	O
al	O
.	O
,	O
2018;Peters	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
cloze	O
modeling	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019;Baevski	O
et	O
al	O
.	O
,	O
2019	O
;	O
have	O
proven	O
to	O
be	O
effective	O
pre	O
-	O
training	O
tasks	O
for	O
NLP	O
.	O
Unlike	O
Electric	O
,	O
these	O
methods	O
follow	O
the	O
standard	O
recipe	O
of	O
estimating	O
token	O
probabilities	O
with	O
an	O
output	O
softmax	O
and	O
using	O
maximumlikelihood	O
training	O
.	O

Energy	O
-	O
based	O
models	O
have	O
been	O
widely	O
explored	O
in	O
machine	O
learning	O
(	O
Dayan	O
et	O
al	O
.	O
,	O
1995	O
et	O
al	O
.	O
,	O
2007	O
)	O
.	O
While	O
many	O
training	O
methods	O
involve	O
sampling	O
from	O
the	O
EBM	O
using	O
gradientbased	O
MCMC	O
(	O
Du	O
and	O
Mordatch	O
,	O
2019	O
)	O
or	O
Gibbs	O
sampling	O
(	O
Hinton	O
,	O
2002	O
)	O
,	O
we	O
considered	O
these	O
approaches	O
too	O
slow	O
for	O
pre	O
-	O
training	O
because	O
they	O
require	O
multiple	O
passes	O
through	O
the	O
model	O
per	O
sample	O
.	O
We	O
instead	O
use	O
noise	O
-	O
contrastive	O
estimation	O
(	O
Gutmann	O
and	O
Hyvärinen	O
,	O
2010	O
)	O
,	O
which	O
has	O
widely	O
been	O
used	O
in	O
NLP	O
for	O
learning	O
word	O
vectors	O
(	O
Mnih	O
and	O
Kavukcuoglu	O
,	O
2013	O
)	O
and	O
text	O
generation	O
models	O
(	O
Jean	O
et	O
al	O
.	O
,	O
2014;Józefowicz	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
While	O
EBMs	O
have	O
previously	O
been	O
applied	O
to	O
leftto	O
-	O
right	O
(	O
Wang	O
et	O
al	O
.	O
,	O
2015	O
)	O
or	O
globally	O
normalized	O
(	O
Rosenfeld	O
et	O
al	O
.	O
,	O
2001;Deng	O
et	O
al	O
.	O
,	O
2020	O
)	O
text	O
generation	O
,	O
they	O
have	O
not	O
previously	O
been	O
applied	O
to	O
cloze	O
models	O
or	O
for	O
pre	O
-	O
training	O
NLP	O
models	O
.	O
Several	O
papers	O
have	O
pointed	O
out	O
the	O
connection	O
between	O
EBMs	O
and	O
GANs	O
(	O
Zhao	O
et	O
al	O
.	O
,	O
2016;Finn	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
which	O
is	O
similar	O
to	O
the	O
Electric	O
/	O
ELECTRA	O
connection	O
.	O

Conclusion	O

We	O
have	O
developed	O
an	O
energy	O
-	O
based	O
cloze	O
model	O
we	O
call	O
Electric	O
and	O
designed	O
an	O
efficient	O
training	O
algorithm	O
for	O
Electric	O
based	O
on	O
noise	O
-	O
contrastive	O
estimation	O
.	O
Although	O
Electric	O
can	O
be	O
derived	O
solely	O
from	O
the	O
cloze	O
task	O
,	O
the	O
resulting	O
pre	O
-	O
training	O
method	O
is	O
closely	O
related	O
to	O
ELECTRA	O
's	O
GANlike	O
pre	O
-	O
training	O
algorithm	O
.	O
While	O
slightly	O
underperforming	O
ELECTRA	O
on	O
downstream	O
tasks	O
,	O
Electric	O
is	O
useful	O
for	O
its	O
ability	O
to	O
quickly	O
produce	O
pseudo	O
-	O
log	O
-	O
likelihood	O
scores	O
for	O
text	O
.	O
Furthermore	O
,	O
it	O
offers	O
a	O
clearer	O
and	O
more	O
principled	O
view	O
of	O
the	O
ELECTRA	O
objective	O
as	O
a	O
"	O
negative	O
sampling	O
"	O
version	O
of	O
cloze	O
pre	O
-	O
training	O
.	O

as	O
ELECTRA	O
's	O
(	O
Clark	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
which	O
adds	O
some	O
additional	O
ideas	O
from	O
on	O
top	O
of	O
the	O
BERT	O
codebase	O
,	O
such	O
as	O
dynamic	O
masking	O
and	O
removing	O
the	O
next	O
-	O
sentence	O
prediction	O
task	O
.	O
We	O
use	O
the	O
weight	O
sharing	O
trick	O
from	O
ELECTRA	O
,	O
where	O
the	O
transformers	O
producing	O
the	O
proposal	O
distribution	O
and	O
the	O
main	O
transformer	O
share	O
token	O
embeddings	O
.	O
We	O
do	O
not	O
use	O
whole	O
-	O
word	O
or	O
n	O
-	O
gram	O
masking	O
,	O
although	O
we	O
believe	O
it	O
would	O
improve	O
results	O
too	O
.	O
We	O
did	O
no	O
hyperparameter	O
tuning	O
,	O
directly	O
using	O
the	O
hyperparameters	O
from	O
ELECTRA	O
-	O
Base	O
for	O
Electric	O
and	O
our	O
baselines	O
.	O
These	O
hyperparameters	O
are	O
slightly	O
modified	O
from	O
the	O
ones	O
used	O
in	O
BERT	O
;	O
for	O
completeness	O
,	O
we	O
show	O
these	O
values	O
in	O
Table	O
3	O
.	O
The	O
hidden	O
sizes	O
,	O
feed	O
-	O
forward	O
hidden	O
sizes	O
,	O
and	O
number	O
of	O
attention	O
heads	O
of	O
the	O
two	O
transformers	O
T	O
LTR	O
and	O
T	O
RTL	O
used	O
to	O
produce	O
the	O
proposal	O
distribution	O
are	O
1/4	O
the	O
size	O
of	O
Electric	O
.	O
We	O
chose	O
this	O
value	O
because	O
it	O
keeps	O
the	O
compute	O
comparable	O
to	O
ELECTRA	O
;	O
running	O
two	O
1/4	O
-	O
sized	O
transformers	O
takes	O
roughly	O
the	O
same	O
compute	O
as	O
running	O
one	O
1/3sized	O
transformer	O
,	O
which	O
is	O
the	O
size	O
of	O
ELECTRA	O
's	O
generator	O
.	O
To	O
make	O
the	O
compute	O
exactly	O
equal	O
,	O
we	O
train	O
Electric	O
for	O
slightly	O
fewer	O
steps	O
than	O
ELEC	O
-	O
TRA	O
.	O
This	O
same	O
generator	O
architecture	O
was	O
used	O
for	O
ELECTRA	O
-	O
TT	O
.	O
The	O
TwoTower	O
baseline	O
consists	O
of	O
two	O
transformers	O
2/3	O
the	O
size	O
of	O
BERT	O
's	O
,	O
which	O
takes	O
approximately	O
the	O
same	O
compute	O
to	O
run	O
.	O
The	O
Electric	O
models	O
,	O
ELECTRA	O
-	O
Base	O
,	O
and	O
BERT	O
-	O
Base	O
all	O
use	O
the	O
same	O
amount	O
of	O
pre	O
-	O
train	O
compute	O
(	O
e.g.	O
,	O
Electric	O
is	O
trained	O
for	O
fewer	O
steps	O
than	O
BERT	O
due	O
to	O
the	O
extra	O
compute	O
from	O
the	O
proposal	O
distribution	O
)	O
,	O
which	O
equates	O
to	O
approximately	O
three	O
days	O
of	O
training	O
on	O
16	O
TPUv2s	O
.	O

B	O
Fine	O
-	O
Tuning	O
Details	O

We	O
use	O
ELECTRA	O
's	O
top	O
-	O
level	O
classifiers	O
and	O
hyperparameter	O
values	O
for	O
fine	O
-	O
tuning	O
as	O
well	O
.	O
For	O
GLUE	O
tasks	O
,	O
a	O
simple	O
linear	O
classifier	O
is	O
added	O
on	O
top	O
of	O
the	O
pre	O
-	O
trained	O
transformer	O
.	O
For	O
SQuAD	O
,	O
a	O
question	O
answering	O
module	O
similar	O
XLNet	O
's	O
(	O
Yang	O
et	O
al	O
.	O
,	O
2019	O
)	O
is	O
added	O
on	O
top	O
of	O
the	O
transformer	O
,	O
which	O
is	O
slightly	O
more	O
sophisticated	O
than	O
BERT	O
's	O
in	O
that	O
it	O
jointly	O
rather	O
than	O
independently	O
predicts	O
the	O
start	O
and	O
end	O
positions	O
and	O
has	O
an	O
"	O
answerability	O
"	O
classifier	O
added	O
for	O
SQuAD	O
2.0	O
.	O
ELECTRA	O
's	O
hyperparameters	O
are	O
similar	O
to	O
BERT	O
's	O
,	O
with	O
the	O
main	O
difference	O
being	O
the	O
addition	O
of	O
a	O
layer	O
-	O
wise	O
learning	O
rate	O
decay	O
where	O
layers	O
of	O
the	O
network	O
closer	O
to	O
the	O
output	O
have	O
a	O
higher	O
learning	O
rate	O
.	O

Following	O
BERT	O
,	O
we	O
submit	O
the	O
best	O
of	O
10	O
models	O
fine	O
-	O
tuned	O
with	O
different	O
random	O
seeds	O
to	O
the	O
GLUE	O
leaderboard	O
for	O
test	O
set	O
results	O
.	O

C	O
Dataset	O
Details	O

We	O
provide	O
details	O
on	O
the	O
fine	O
-	O
tuning	O
datasets	O
below	O
.	O
All	O
datasets	O
are	O
in	O
English	O
.	O
GLUE	O
data	O
can	O
be	O
downloaded	O
at	O
https://	O
gluebenchmark.com/	O
and	O
SQuAD	O
data	O
can	O
be	O
downloaded	O
at	O
https://rajpurkar	O
.	O
github.io/SQuAD-explorer/.	O

•	O
CoLA	O
:	O
Corpus	O
of	O
Linguistic	O
Acceptability	O
(	O
Warstadt	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
The	O
task	O
is	O
to	O
determine	O
whether	O
a	O
given	O
sentence	O
is	O
grammatical	O
or	O
not	O
.	O
The	O
dataset	O
contains	O
8.5k	O
train	O
examples	O
from	O
books	O
and	O
journal	O
articles	O
on	O
linguistic	O
theory	O
.	O

•	O
SST	O
:	O
Stanford	O
Sentiment	O
Treebank	O
(	O
Socher	O
et	O
al	O
.	O
,	O
2013	O
)	O
.	O
The	O
tasks	O
is	O
to	O
determine	O
if	O
the	O
sentence	O
is	O
positive	O
or	O
negative	O
in	O
sentiment	O
.	O

The	O
dataset	O
contains	O
67k	O
train	O
examples	O
from	O
movie	O
reviews	O
.	O

•	O
MRPC	O
:	O
Microsoft	O
Research	O
Paraphrase	O
Corpus	O
(	O
Dolan	O
and	O
Brockett	O
,	O
2005	O
)	O
.	O
The	O
task	O
is	O
to	O
predict	O
whether	O
two	O
sentences	O
are	O
semantically	O
equivalent	O
or	O
not	O
.	O
The	O
dataset	O
contains	O
3.7k	O
train	O
examples	O
from	O
online	O
news	O
sources	O
.	O

•	O
STS	O
:	O
Semantic	O
Textual	O
Similarity	O
(	O
Cer	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
The	O
tasks	O
is	O
to	O
predict	O
how	O
semantically	O
similar	O
two	O
sentences	O
are	O
on	O
a	O
1	O
-	O
5	O
scale	O
.	O
The	O
dataset	O
contains	O
5.8k	O
train	O
examples	O
drawn	O
from	O
new	O
headlines	O
,	O
video	O
and	O
image	O
captions	O
,	O
and	O
natural	O
language	O
inference	O
data	O
.	O

•	O
QQP	O
:	O
Quora	O
Question	O
Pairs	O
(	O
Iyer	O
et	O
al	O
.	O
,	O
2017	O
)	O
.	O
The	O
task	O
is	O
to	O
determine	O
whether	O
a	O
pair	O
of	O
questions	O
are	O
semantically	O
equivalent	O
.	O
The	O
dataset	O
contains	O
364k	O
train	O
examples	O
from	O
the	O
community	O
question	O
-	O
answering	O
website	O
Quora	O
.	O

•	O
MNLI	O
:	O
Multi	O
-	O
genre	O
Natural	O
Language	O
Inference	O
(	O
Williams	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
Given	O
a	O
premise	O
sentence	O
and	O
a	O
hypothesis	O
sentence	O
,	O
the	O
task	O
is	O
to	O
predict	O
whether	O
the	O
premise	O
entails	O
the	O
hypothesis	O
,	O
contradicts	O
the	O
hypothesis	O
,	O
or	O
neither	O
.	O
The	O
dataset	O
contains	O
393k	O
train	O
examples	O
drawn	O
from	O
ten	O
different	O
sources	O
.	O
using	O
dev	O
-	O
set	O
model	O
selection	O
to	O
choose	O
the	O
test	O
set	O
submission	O
may	O
alleviate	O
the	O
high	O
variance	O
of	O
fine	O
-	O
tuning	O
to	O
some	O
extent	O
,	O
such	O
model	O
selection	O
is	O
still	O
not	O
sufficient	O
for	O
reliable	O
comparisons	O
between	O
methods	O
(	O
Reimers	O
and	O
Gurevych	O
,	O
2018	O
)	O
.	O

Acknowledgements	O

We	O
thank	O
John	O
Hewitt	O
,	O
Yuhao	O
Zhang	O
,	O
Ashwin	O
Paranjape	O
,	O
Sergey	O
Levine	O
,	O
and	O
the	O
anonymous	O
reviewers	O
for	O
their	O
thoughtful	O
comments	O
and	O
suggestions	O
.	O
Kevin	O
is	O
supported	O
by	O
a	O
Google	O
PhD	O
Fellowship	O
.	O

A	O
Pre	O
-	O
Training	O
Details	O

The	O
neural	O
architectures	O
of	O
our	O
models	O
are	O
identical	O
to	O
BERT	O
-	O
Base	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
although	O
we	O
believe	O
incorporating	O
additions	O
such	O
as	O
relative	O
position	O
encodings	O
(	O
Shaw	O
et	O
al	O
.	O
,	O
2018	O
)	O
•	O
QNLI	O
:	O
Question	O
Natural	O
Language	O
Inference	O
;	O
constructed	O
from	O
SQuAD	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
The	O
task	O
is	O
to	O
predict	O
whether	O
a	O
context	O
sentence	O
contains	O
the	O
answer	O
to	O
a	O
question	O
sentence	O
.	O
The	O
dataset	O
contains	O
108k	O
train	O
examples	O
from	O
Wikipedia	O
.	O

•	O
RTE	O
:	O
Recognizing	O
Textual	O
Entailment	O
(	O
Giampiccolo	O
et	O
al	O
.	O
,	O
2007	O
)	O
.	O
Given	O
a	O
premise	O
sentence	O
and	O
a	O
hypothesis	O
sentence	O
,	O
the	O
task	O
is	O
to	O
predict	O
whether	O
the	O
premise	O
entails	O
the	O
hypothesis	O
or	O
not	O
.	O
The	O
dataset	O
contains	O
2.5k	O
train	O
examples	O
from	O
a	O
series	O
of	O
annual	O
textual	O
entailment	O
challenges	O
.	O

•	O
SQuAD	O
1.1	O
:	O
Stanford	O
Question	O
Answering	O
Dataset	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
Given	O
a	O
context	O
paragraph	O
and	O
a	O
question	O
,	O
the	O
task	O
is	O
to	O
select	O
the	O
span	O
of	O
text	O
in	O
the	O
paragraph	O
answering	O
the	O
question	O
.	O
The	O
dataset	O
contains	O
88k	O
train	O
examples	O
from	O
Wikipedia	O
.	O

•	O
SQuAD	O
2.0	O
:	O
Stanford	O
Question	O
Answering	O
Dataset	O
version	O
2.0	O
(	O
Rajpurkar	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
This	O
task	O
adds	O
addition	O
questions	O
to	O
SQuAD	O
whose	O
answer	O
does	O
not	O
exist	O
in	O
the	O
context	O
;	O
models	O
have	O
to	O
recognize	O
when	O
these	O
questions	O
occur	O
and	O
not	O
return	O
an	O
answer	O
for	O
them	O
.	O

The	O
dataset	O
contains	O
130k	O
train	O
examples	O
,	O

We	O
report	O
Spearman	O
correlation	O
for	O
STS	O
,	O
Matthews	O
correlation	O
coefficient	O
(	O
MCC	O
)	O
for	O
CoLA	O
,	O
exact	O
match	O
for	O
SQuAD	O
,	O
and	O
accuracy	O
for	O
the	O
other	O
tasks	O
.	O
We	O
use	O
the	O
provided	O
evaluation	O
script	O
for	O
SQuAD	O
6	O
,	O
scipy	O
to	O
compute	O
Spearman	O
scores	O
7	O
,	O
and	O
sklearn	O
to	O
compute	O
MCC	O
8	O
.	O
We	O
use	O
the	O
standard	O
train	O
/	O
dev	O
/	O
test	O
splits	O
.	O

D	O
Detailed	O
Results	O

We	O
show	O
detailed	O
results	O
on	O
GLUE	O
and	O
SQuAD	O
in	O
Table	O
4	O
and	O
detailed	O
results	O
on	O
LibriSpeech	O
reranking	O
in	O
Table	O
5	O
.	O
Following	O
BERT	O
,	O
we	O
do	O
not	O
show	O
results	O
on	O
the	O
WNLI	O
GLUE	O
task	O
,	O
as	O
it	O
is	O
difficult	O
to	O
beat	O
even	O
the	O
majority	O
classifier	O
using	O
a	O
standard	O
fine	O
-	O
tuning	O
-	O
as	O
-	O
classifier	O
approach	O
.	O
We	O
show	O
dev	O
rather	O
than	O
test	O
results	O
on	O
GLUE	O
in	O
the	O
main	O
paper	O
because	O
they	O
are	O
more	O
reliable	O
;	O
the	O
performance	O
of	O
fine	O
-	O
tuned	O
models	O
varies	O
substantially	O
based	O
on	O
the	O
random	O
seed	O
(	O
Phang	O
et	O
al	O
.	O
,	O
2018;Clark	O
et	O
al	O
.	O
,	O
2019;Dodge	O
et	O
al	O
.	O
,	O
2020	O
)	O
,	O
but	O
GLUE	O
only	O
supports	O
submitting	O
a	O
single	O
model	O
rather	O
than	O
getting	O
a	O
median	O
score	O
of	O
multiple	O
models	O
.	O
While	O
6	O
https://worksheets	O
.	O
codalab.org/rest/bundles/	O
0x6b567e1cf2e041ec80d7098f031c5c9e/	O
contents	O
/	O
blob/	O
7	O
https://docs.scipy.org/doc/	O
scipy	O
/	O
reference	O
/	O
generated	O
/	O
scipy.stats	O
.	O
spearmanr.html	O

FIND	O
:	O
Human	O
-	O
in	O
-	O
the	O
-	O
Loop	O
Debugging	O
Deep	O
Text	O
Classifiers	O

Since	O
obtaining	O
a	O
perfect	O
training	O
dataset	O
(	O
i.e.	O
,	O
a	O
dataset	O
which	O
is	O
considerably	O
large	O
,	O
unbiased	O
,	O
and	O
well	O
-	O
representative	O
of	O
unseen	O
cases	O
)	O
is	O
hardly	O
possible	O
,	O
many	O
real	O
-	O
world	O
text	O
classifiers	O
are	O
trained	O
on	O
the	O
available	O
,	O
yet	O
imperfect	O
,	O
datasets	O
.	O
These	O
classifiers	O
are	O
thus	O
likely	O
to	O
have	O
undesirable	O
properties	O
.	O
For	O
instance	O
,	O
they	O
may	O
have	O
biases	O
against	O
some	O
sub	O
-	O
populations	O
or	O
may	O
not	O
work	O
effectively	O
in	O
the	O
wild	O
due	O
to	O
overfitting	O
.	O
In	O
this	O
paper	O
,	O
we	O
propose	O
FINDa	O
framework	O
which	O
enables	O
humans	O
to	O
debug	O
deep	O
learning	O
text	O
classifiers	O
by	O
disabling	O
irrelevant	O
hidden	O
features	O
.	O
Experiments	O
show	O
that	O
by	O
using	O
FIND	O
,	O
humans	O
can	O
improve	O
CNN	O
text	O
classifiers	O
which	O
were	O
trained	O
under	O
different	O
types	O
of	O
imperfect	O
datasets	O
(	O
including	O
datasets	O
with	O
biases	O
and	O
datasets	O
with	O
dissimilar	O
traintest	O
distributions	O
)	O
.	O

Introduction	O

Deep	O
learning	O
has	O
become	O
the	O
dominant	O
approach	O
to	O
address	O
most	O
Natural	O
Language	O
Processing	O
(	O
NLP	O
)	O
tasks	O
,	O
including	O
text	O
classification	O
.	O
With	O
sufficient	O
and	O
high	O
-	O
quality	O
training	O
data	O
,	O
deep	O
learning	O
models	O
can	O
perform	O
incredibly	O
well	O
.	O
However	O
,	O
in	O
real	O
-	O
world	O
cases	O
,	O
such	O
ideal	O
datasets	O
are	O
scarce	O
.	O
Often	O
times	O
,	O
the	O
available	O
datasets	O
are	O
small	O
,	O
full	O
of	O
regular	O
but	O
irrelevant	O
words	O
,	O
and	O
contain	O
unintended	O
biases	O
(	O
Wiegand	O
et	O
al	O
.	O
,	O
2019;Gururangan	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
These	O
can	O
lead	O
to	O
suboptimal	O
models	O
with	O
undesirable	O
properties	O
.	O
For	O
example	O
,	O
the	O
models	O
may	O
have	O
biases	O
against	O
some	O
sub	O
-	O
populations	O
or	O
may	O
not	O
work	O
effectively	O
in	O
the	O
wild	O
as	O
they	O
overfit	O
the	O
imperfect	O
training	O
data	O
.	O

To	O
improve	O
the	O
models	O
,	O
previous	O
work	O
has	O
looked	O
into	O
different	O
techniques	O
beyond	O
standard	O
model	O
fitting	O
.	O
If	O
the	O
weaknesses	O
of	O
the	O
training	O
datasets	O
or	O
the	O
models	O
are	O
anticipated	O
,	O
strategies	O
can	O
be	O
tailored	O
to	O
mitigate	O
such	O
weaknesses	O
.	O
For	O
example	O
,	O
augmenting	O
the	O
training	O
data	O
with	O
genderswapped	O
input	O
texts	O
helps	O
reduce	O
gender	O
bias	O
in	O
the	O
models	O
(	O
Park	O
et	O
al	O
.	O
,	O
2018;Zhao	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
Adversarial	O
training	O
can	O
prevent	O
the	O
models	O
from	O
exploiting	O
irrelevant	O
and/or	O
protected	O
features	O
(	O
Jaiswal	O
et	O
al	O
.	O
,	O
2019;Zhang	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
With	O
a	O
limited	O
number	O
of	O
training	O
examples	O
,	O
using	O
human	O
rationales	O
or	O
prior	O
knowledge	O
together	O
with	O
training	O
labels	O
can	O
help	O
the	O
models	O
perform	O
better	O
(	O
Zaidan	O
et	O
al	O
.	O
,	O
2007;Bao	O
et	O
al	O
.	O
,	O
2018;Liu	O
and	O
Avci	O
,	O
2019	O
)	O
.	O

Nonetheless	O
,	O
there	O
are	O
side	O
-	O
effects	O
of	O
suboptimal	O
datasets	O
that	O
can	O
not	O
be	O
predicted	O
and	O
are	O
only	O
found	O
after	O
training	O
thanks	O
to	O
post	O
-	O
hoc	O
error	O
analysis	O
.	O
To	O
rectify	O
such	O
problems	O
,	O
there	O
have	O
been	O
attempts	O
to	O
enable	O
humans	O
to	O
fix	O
the	O
trained	O
models	O
(	O
i.e.	O
,	O
to	O
perform	O
model	O
debugging	O
)	O
(	O
Stumpf	O
et	O
al	O
.	O
,	O
2009;Teso	O
and	O
Kersting	O
,	O
2019	O
)	O
.	O
Since	O
the	O
models	O
are	O
usually	O
too	O
complex	O
to	O
understand	O
,	O
manually	O
modifying	O
the	O
model	O
parameters	O
is	O
not	O
possible	O
.	O
Existing	O
techniques	O
,	O
therefore	O
,	O
allow	O
humans	O
to	O
provide	O
feedback	O
on	O
individual	O
predictions	O
instead	O
.	O
Then	O
,	O
additional	O
training	O
examples	O
are	O
created	O
based	O
on	O
the	O
feedback	O
to	O
retrain	O
the	O
models	O
.	O
However	O
,	O
such	O
local	O
improvements	O
for	O
individual	O
predictions	O
could	O
add	O
up	O
to	O
inferior	O
overall	O
performance	O
(	O
Wu	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Furthermore	O
,	O
these	O
existing	O
techniques	O
allow	O
us	O
to	O
rectify	O
only	O
errors	O
related	O
to	O
examples	O
at	O
hand	O
but	O
provide	O
no	O
way	O
to	O
fix	O
problems	O
kept	O
hidden	O
in	O
the	O
model	O
parameters	O
.	O

In	O
this	O
paper	O
,	O
we	O
propose	O
a	O
framework	O
which	O
allows	O
humans	O
to	O
debug	O
and	O
improve	O
deep	O
text	O
classifiers	O
by	O
disabling	O
hidden	O
features	O
which	O
are	O
irrelevant	O
to	O
the	O
classification	O
task	O
.	O
We	O
name	O
this	O
framework	O
FIND	O
(	O
Feature	O
Investigation	O
aNd	O
Disabling	O
)	O
.	O
FIND	O
exploits	O
an	O
explanation	O
method	O
,	O
namely	O
layer	O
-	O
wise	O
relevance	O
propagation	O
(	O
LRP	O
)	O
(	O
Arras	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
to	O
understand	O
the	O
behavior	O
of	O
a	O
classifier	O
when	O
it	O
predicts	O
each	O
training	O
instance	O
.	O

Then	O
it	O
aggregates	O
all	O
the	O
information	O
using	O
word	O
clouds	O
to	O
create	O
a	O
global	O
visual	O
picture	O
of	O
the	O
model	O
.	O
This	O
enables	O
humans	O
to	O
comprehend	O
the	O
features	O
automatically	O
learned	O
by	O
the	O
deep	O
classifier	O
and	O
then	O
decide	O
to	O
disable	O
some	O
features	O
that	O
could	O
undermine	O
the	O
prediction	O
accuracy	O
during	O
testing	O
.	O
The	O
main	O
differences	O
between	O
our	O
work	O
and	O
existing	O
work	O
are	O
:	O
(	O
i	O
)	O
first	O
,	O
FIND	O
leverages	O
human	O
feedback	O
on	O
the	O
model	O
components	O
,	O
not	O
the	O
individual	O
predictions	O
,	O
to	O
perform	O
debugging	O
;	O
(	O
ii	O
)	O
second	O
,	O
FIND	O
targets	O
deep	O
text	O
classifiers	O
which	O
are	O
more	O
convoluted	O
than	O
traditional	O
classifiers	O
used	O
in	O
existing	O
work	O
(	O
such	O
as	O
Naive	O
Bayes	O
classifiers	O
and	O
Support	O
Vector	O
Machines	O
)	O
.	O

We	O
conducted	O
three	O
human	O
experiments	O
(	O
one	O
feasibility	O
study	O
and	O
two	O
debugging	O
experiments	O
)	O
to	O
demonstrate	O
the	O
usefulness	O
of	O
FIND	O
.	O
For	O
all	O
the	O
experiments	O
,	O
we	O
used	O
as	O
classifiers	O
convolutional	O
neural	O
networks	O
(	O
CNNs	O
)	O
(	O
Kim	O
,	O
2014	O
)	O
,	O
which	O
are	O
a	O
popular	O
,	O
well	O
-	O
performing	O
architecture	O
for	O
many	O
text	O
classification	O
tasks	O
including	O
the	O
tasks	O
we	O
experimented	O
with	O
(	O
Gambäck	O
and	O
Sikdar	O
,	O
2017;Johnson	O
and	O
Zhang	O
,	O
2015;Zhang	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
The	O
overall	O
results	O
show	O
that	O
FIND	O
with	O
human	O
-	O
in	O
-	O
the	O
-	O
loop	O
can	O
improve	O
the	O
text	O
classifiers	O
and	O
mitigate	O
the	O
said	O
problems	O
in	O
the	O
datasets	O
.	O
After	O
the	O
experiments	O
,	O
we	O
discuss	O
the	O
generalization	O
of	O
the	O
proposed	O
framework	O
to	O
other	O
tasks	O
and	O
models	O
.	O
Overall	O
,	O
the	O
main	O
contributions	O
of	O
this	O
paper	O
are	O
:	O

•	O
We	O
propose	O
using	O
word	O
clouds	O
as	O
visual	O
explanations	O
of	O
the	O
features	O
learned	O
.	O

•	O
We	O
propose	O
a	O
technique	O
to	O
disable	O
the	O
learned	O
features	O
which	O
are	O
irrelevant	O
or	O
harmful	O
to	O
the	O
classification	O
task	O
so	O
as	O
to	O
improve	O
the	O
classifier	O
.	O
This	O
technique	O
and	O
the	O
word	O
clouds	O
form	O
the	O
human	O
-	O
debugging	O
framework	O
-FIND	O
.	O

•	O
We	O
conduct	O
three	O
human	O
experiments	O
that	O
demonstrate	O
the	O
effectiveness	O
of	O
FIND	O
in	O
different	O
scenarios	O
.	O
The	O
results	O
not	O
only	O
highlight	O
the	O
usefulness	O
of	O
our	O
approach	O
but	O
also	O
reveal	O
interesting	O
behaviors	O
of	O
CNNs	O
for	O
text	O
classification	O
.	O

The	O
rest	O
of	O
this	O
paper	O
is	O
organized	O
as	O
follows	O
.	O
Section	O
2	O
explains	O
related	O
work	O
about	O
analyzing	O
,	O
explaining	O
,	O
and	O
human	O
-	O
debugging	O
text	O
classifiers	O
.	O
Section	O
3	O
proposes	O
FIND	O
,	O
our	O
debugging	O
framework	O
.	O
Section	O
4	O
explains	O
the	O
experimental	O
setup	O
followed	O
by	O
the	O
three	O
human	O
experiments	O
in	O
Section	O
5	O
to	O
7	O
.	O
Finally	O
,	O
Section	O
8	O
discusses	O
generalization	O
of	O
the	O
framework	O
and	O
concludes	O
the	O
paper	O
.	O
Code	O
and	O
datasets	O
of	O
this	O
paper	O
are	O
available	O
at	O
https://github.com/plkumjorn/FIND	O
.	O

Related	O
Work	O

Analyzing	O
deep	O
NLP	O
models	O
-There	O
has	O
been	O
substantial	O
work	O
in	O
gaining	O
better	O
understanding	O
of	O
complex	O
,	O
deep	O
neural	O
NLP	O
models	O
.	O
By	O
visualizing	O
dense	O
hidden	O
vectors	O
,	O
Li	O
et	O
al	O
.	O
(	O
2016	O
)	O
found	O
that	O
some	O
dimensions	O
of	O
the	O
final	O
representation	O
learned	O
by	O
recurrent	O
neural	O
networks	O
capture	O
the	O
effect	O
of	O
intensification	O
and	O
negation	O
in	O
the	O
input	O
text	O
.	O
Karpathy	O
et	O
al	O
.	O
(	O
2015	O
)	O
revealed	O
the	O
existence	O
of	O
interpretable	O
cells	O
in	O
a	O
character	O
-	O
level	O
LSTM	O
model	O
for	O
language	O
modelling	O
.	O
For	O
example	O
,	O
they	O
found	O
a	O
cell	O
acting	O
as	O
a	O
line	O
length	O
counter	O
and	O
cells	O
checking	O
if	O
the	O
current	O
letter	O
is	O
inside	O
a	O
parenthesis	O
or	O
a	O
quote	O
.	O
Jacovi	O
et	O
al	O
.	O
(	O
2018	O
)	O
presented	O
interesting	O
findings	O
about	O
CNNs	O
for	O
text	O
classification	O
including	O
the	O
fact	O
that	O
one	O
convolutional	O
filter	O
may	O
detect	O
more	O
than	O
one	O
n	O
-	O
gram	O
pattern	O
and	O
may	O
also	O
suppress	O
negative	O
n	O
-	O
grams	O
.	O
Many	O
recent	O
papers	O
studied	O
several	O
types	O
of	O
knowledge	O
in	O
BERT	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
,	O
a	O
deep	O
transformer	O
-	O
based	O
model	O
for	O
language	O
understanding	O
,	O
and	O
found	O
that	O
syntactic	O
information	O
is	O
mostly	O
captured	O
in	O
the	O
middle	O
BERT	O
layers	O
while	O
the	O
final	O
BERT	O
layers	O
are	O
the	O
most	O
task	O
-	O
specific	O
(	O
Rogers	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
Inspired	O
by	O
many	O
findings	O
,	O
we	O
make	O
the	O
assumption	O
that	O
each	O
dimension	O
of	O
the	O
final	O
representation	O
(	O
i.e.	O
,	O
the	O
vector	O
before	O
the	O
output	O
layer	O
)	O
captures	O
patterns	O
or	O
qualities	O
in	O
the	O
input	O
which	O
are	O
useful	O
for	O
classification	O
.	O
Therefore	O
,	O
understanding	O
the	O
roles	O
of	O
these	O
dimensions	O
(	O
we	O
refer	O
to	O
them	O
as	O
features	O
)	O
is	O
a	O
prerequisite	O
for	O
effective	O
human	O
-	O
in	O
-	O
the	O
-	O
loop	O
model	O
debugging	O
,	O
and	O
we	O
exploit	O
an	O
explanation	O
method	O
to	O
gain	O
such	O
an	O
understanding	O
.	O
Explaining	O
predictions	O
from	O
text	O
classifiers	O
-Several	O
methods	O
have	O
been	O
devised	O
to	O
generate	O
explanations	O
supporting	O
classifications	O
in	O
many	O
forms	O
,	O
such	O
as	O
natural	O
language	O
texts	O
,	O
rules	O
(	O
Ribeiro	O
et	O
al	O
.	O
,	O
2018	O
)	O
,	O
extracted	O
rationales	O
(	O
Lei	O
et	O
al	O
.	O
,	O
2016	O
)	O
,	O
and	O
attribution	O
scores	O
(	O
Lertvittayakumjorn	O
and	O
Toni	O
,	O
2019	O
)	O
.	O
Some	O
explanation	O
methods	O
,	O
such	O
as	O
LIME	O
(	O
Ribeiro	O
et	O
al	O
.	O
,	O
2016	O
)	O
and	O
SHAP	O
(	O
Lundberg	O
and	O
Lee	O
,	O
2017	O
)	O
,	O
are	O
model	O
-	O
agnostic	O
and	O
do	O
not	O
require	O
access	O
to	O
model	O
parameters	O
.	O
Other	O
methods	O
access	O
the	O
model	O
architectures	O
and	O
parameters	O
to	O
generate	O
the	O
explanations	O
,	O
such	O
as	O
DeepLIFT	O
(	O
Shrikumar	O
et	O
al	O
.	O
,	O
2017	O
)	O
and	O
LRP	O
(	O
layer	O
-	O
wise	O
relevance	O
propagation	O
)	O
(	O
Bach	O
et	O
al	O
.	O
,	O
2015;Arras	O
et	O
al	O
.	O
,	O
2016	O
)	O
.	O
In	O
this	O
work	O
,	O
we	O
use	O
LRP	O
to	O
explain	O
not	O
the	O
predictions	O
but	O
the	O
learned	O
features	O
so	O
as	O
to	O
expose	O
the	O
model	O
behavior	O
to	O
humans	O
and	O
enable	O
informed	O
model	O
debugging	O
.	O

Debugging	O
text	O
classifiers	O
using	O
human	O
feedback	O
-Early	O
work	O
in	O
this	O
area	O
comes	O
from	O
the	O
human	O
-	O
computer	O
interaction	O
community	O
.	O
Stumpf	O
et	O
al	O
.	O
(	O
2009	O
)	O
studied	O
the	O
types	O
of	O
feedback	O
humans	O
usually	O
give	O
in	O
response	O
to	O
machine	O
-	O
generated	O
predictions	O
and	O
explanations	O
.	O
Also	O
,	O
some	O
of	O
the	O
feedback	O
collected	O
(	O
i.e.	O
,	O
important	O
words	O
of	O
each	O
category	O
)	O
was	O
used	O
to	O
improve	O
the	O
classifier	O
via	O
a	O
user	O
co	O
-	O
training	O
approach	O
.	O
Kulesza	O
et	O
al	O
.	O
(	O
2015	O
)	O
presented	O
an	O
explanatory	O
debugging	O
approach	O
in	O
which	O
the	O
system	O
explains	O
to	O
users	O
how	O
it	O
made	O
each	O
prediction	O
,	O
and	O
the	O
users	O
then	O
rectify	O
the	O
model	O
by	O
adding	O
/	O
removing	O
words	O
from	O
the	O
explanation	O
and	O
adjusting	O
important	O
weights	O
.	O
Even	O
without	O
explanations	O
shown	O
,	O
an	O
active	O
learning	O
framework	O
proposed	O
by	O
Settles	O
(	O
2011	O
)	O
asks	O
humans	O
to	O
iteratively	O
label	O
some	O
chosen	O
features	O
(	O
i.e.	O
,	O
words	O
)	O
and	O
adjusts	O
the	O
model	O
parameters	O
that	O
correspond	O
to	O
the	O
features	O
.	O
However	O
,	O
these	O
early	O
works	O
target	O
simpler	O
machine	O
learning	O
classifiers	O
(	O
e.g.	O
,	O
Naive	O
Bayes	O
classifiers	O
with	O
bag	O
-	O
of	O
-	O
words	O
)	O
and	O
it	O
is	O
not	O
clear	O
how	O
to	O
apply	O
the	O
proposed	O
approaches	O
to	O
deep	O
text	O
classifiers	O
.	O

Recently	O
,	O
there	O
have	O
been	O
new	O
attempts	O
to	O
use	O
explanations	O
and	O
human	O
feedback	O
to	O
debug	O
classifiers	O
in	O
general	O
.	O
Some	O
of	O
them	O
were	O
tested	O
on	O
traditional	O
text	O
classifiers	O
.	O
For	O
instance	O
,	O
Ribeiro	O
et	O
al	O
.	O
(	O
2016	O
)	O
showed	O
a	O
set	O
of	O
LIME	O
explanations	O
for	O
individual	O
SVM	O
predictions	O
to	O
humans	O
and	O
asked	O
them	O
to	O
remove	O
irrelevant	O
words	O
from	O
the	O
training	O
data	O
in	O
subsequent	O
training	O
.	O
The	O
process	O
was	O
run	O
for	O
three	O
rounds	O
to	O
iteratively	O
improve	O
the	O
classifiers	O
.	O
Teso	O
and	O
Kersting	O
(	O
2019	O
)	O
proposed	O
CAIPI	O
,	O
which	O
is	O
an	O
explanatory	O
interactive	O
learning	O
framework	O
.	O
At	O
each	O
iteration	O
,	O
it	O
selects	O
an	O
unlabelled	O
example	O
to	O
predict	O
and	O
explain	O
to	O
users	O
using	O
LIME	O
,	O
and	O
the	O
users	O
respond	O
by	O
removing	O
irrelevant	O
features	O
from	O
the	O
explanation	O
.	O
CAIPI	O
then	O
uses	O
this	O
feedback	O
to	O
generate	O
augmented	O
data	O
and	O
retrain	O
the	O
model	O
.	O
While	O
these	O
recent	O
works	O
use	O
feedback	O
on	O
lowlevel	O
features	O
(	O
input	O
words	O
)	O
and	O
individual	O
predictions	O
,	O
our	O
framework	O
(	O
FIND	O
)	O
uses	O
feedback	O
on	O
the	O
learned	O
features	O
with	O
respect	O
to	O
the	O
big	O
picture	O
of	O
the	O
model	O
.	O
This	O
helps	O
us	O
avoid	O
local	O
decision	O
pitfalls	O
which	O
usually	O
occur	O
in	O
interactive	O
machine	O
learning	O
(	O
Wu	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Overall	O
,	O
what	O
makes	O
our	O
contribution	O
different	O
from	O
existing	O
work	O
is	O
that	O
(	O
i	O
)	O
we	O
collect	O
the	O
feedback	O
on	O
the	O
model	O
,	O
not	O
the	O
individual	O
predictions	O
,	O
and	O
(	O
ii	O
)	O
we	O
target	O
deep	O
text	O
classifiers	O
which	O
are	O
more	O
complex	O
than	O
the	O
models	O
used	O
in	O
previous	O
work	O
.	O

FIND	O
:	O
Debugging	O
Text	O
Classifiers	O

Motivation	O

Generally	O
,	O
deep	O
text	O
classifiers	O
can	O
be	O
divided	O
into	O
two	O
parts	O
.	O
The	O
first	O
part	O
performs	O
feature	O
extraction	O
,	O
transforming	O
an	O
input	O
text	O
into	O
a	O
dense	O
vector	O
(	O
i.e.	O
,	O
a	O
feature	O
vector	O
)	O
which	O
represents	O
the	O
input	O
.	O
There	O
are	O
several	O
alternatives	O
to	O
implement	O
this	O
part	O
such	O
as	O
using	O
convolutional	O
layers	O
,	O
recurrent	O
layers	O
,	O
and	O
transformer	O
layers	O
.	O
The	O
second	O
part	O
performs	O
classification	O
passing	O
the	O
feature	O
vector	O
through	O
a	O
dense	O
layer	O
with	O
softmax	O
activation	O
to	O
get	O
predicted	O
probability	O
of	O
the	O
classes	O
.	O
These	O
deep	O
classifiers	O
are	O
not	O
transparent	O
,	O
as	O
humans	O
can	O
not	O
interpret	O
the	O
meaning	O
of	O
either	O
the	O
intermediate	O
vectors	O
or	O
the	O
model	O
parameters	O
used	O
for	O
feature	O
extraction	O
.	O
This	O
prevents	O
humans	O
from	O
applying	O
their	O
knowledge	O
to	O
modify	O
or	O
debug	O
the	O
classifiers	O
.	O

In	O
contrast	O
,	O
if	O
we	O
understand	O
which	O
patterns	O
or	O
qualities	O
of	O
the	O
input	O
are	O
captured	O
in	O
each	O
feature	O
,	O
we	O
can	O
comprehend	O
the	O
overall	O
reasoning	O
mechanism	O
of	O
the	O
model	O
as	O
the	O
dense	O
layer	O
in	O
the	O
classification	O
part	O
then	O
becomes	O
interpretable	O
.	O
In	O
this	O
paper	O
,	O
we	O
make	O
this	O
possible	O
using	O
LRP	O
.	O
By	O
understanding	O
the	O
model	O
,	O
humans	O
can	O
check	O
whether	O
the	O
input	O
patterns	O
detected	O
by	O
each	O
feature	O
are	O
relevant	O
for	O
classification	O
.	O
Also	O
,	O
the	O
features	O
should	O
be	O
used	O
by	O
the	O
subsequent	O
dense	O
layer	O
to	O
support	O
the	O
right	O
classes	O
.	O
If	O
these	O
are	O
not	O
the	O
case	O
,	O
debugging	O
can	O
be	O
done	O
by	O
disabling	O
the	O
features	O
which	O
may	O
be	O
harmful	O
if	O
they	O
exist	O
in	O
the	O
model	O
.	O
Figure	O
1	O
shows	O
the	O
overview	O
of	O
our	O
debugging	O
framework	O
,	O
FIND	O
.	O

Notation	O

Let	O
us	O
consider	O
a	O
text	O
classification	O
task	O
with	O
|C|	O
classes	O
where	O
C	O
is	O
the	O
set	O
of	O
all	O
classes	O
and	O
let	O
V	O
be	O
a	O
set	O
of	O
unique	O
words	O
in	O
the	O
corpus	O
(	O
the	O
vocabulary	O
)	O
.	O

A	O
training	O
dataset	O
D	O
=	O
{	O
(	O
x	O
1	O
,	O
y	O
1	O
)	O
,	O
.	O
.	O
.	O
,	O
(	O
x	O
N	O
,	O
y	O
N	O
)	O
}	O
is	O
given	O
,	O
where	O
x	O
i	O
is	O
the	O
i	O
-	O
th	O
document	O
containing	O
a	O
sequence	O
of	O
L	O
words	O
,	O
[	O
x	O
i1	O
,	O
x	O
i2	O
,	O
...	O
,	O
x	O
iL	O
]	O
,	O
and	O
y	O
i	O
∈	O
C	O
is	O
the	O
class	O
label	O
of	O

Understanding	O
the	O
Model	O

To	O
understand	O
how	O
the	O
model	O
M	O
works	O
,	O
we	O
analyze	O
the	O
patterns	O
or	O
characteristics	O
of	O
the	O
input	O
that	O
activate	O
each	O
feature	O
f	O
i	O
.	O
Specifically	O
,	O
using	O
LRP	O
1	O
,	O
for	O
each	O
f	O
i	O
of	O
an	O
example	O
x	O
j	O
in	O
the	O
training	O
dataset	O
,	O
we	O
calculate	O
a	O
relevance	O
vector	O
r	O
ij	O
∈	O
R	O
L	O
showing	O
the	O
relevance	O
scores	O
(	O
the	O
contributions	O
)	O
of	O
each	O
word	O
in	O
x	O
j	O
for	O
the	O
value	O
of	O
f	O
i	O
.	O
After	O
doing	O
this	O
for	O
all	O
d	O
features	O
of	O
all	O
training	O
examples	O
,	O
we	O
can	O
produce	O
word	O
clouds	O
to	O
help	O
the	O
users	O
better	O
understand	O
the	O
model	O
M	O
.	O
Word	O
clouds	O
-For	O
each	O
feature	O
f	O
i	O
,	O
we	O
create	O
(	O
one	O
or	O
more	O
)	O
word	O
clouds	O
to	O
visualize	O
the	O
patterns	O
in	O
the	O
input	O
texts	O
which	O
highly	O
activate	O
f	O
i	O
.	O
This	O
can	O
be	O
done	O
by	O
analyzing	O
r	O
ij	O
for	O
all	O
x	O
j	O
in	O
the	O
training	O
data	O
and	O
displaying	O
,	O
in	O
the	O
word	O
clouds	O
,	O
words	O
or	O
n	O
-	O
grams	O
which	O
get	O
high	O
relevance	O
scores	O
.	O
Note	O
that	O
different	O
model	O
architectures	O
may	O
have	O
different	O
ways	O
to	O
generate	O
the	O
word	O
clouds	O
so	O
as	O
to	O
effectively	O
reveal	O
the	O
behavior	O
of	O
the	O
features	O
.	O

For	O
CNNs	O
,	O
the	O
classifiers	O
we	O
experiment	O
with	O
in	O
this	O
paper	O
,	O
each	O
feature	O
has	O
one	O
word	O
cloud	O
containing	O
the	O
n	O
-	O
grams	O
,	O
from	O
the	O
training	O
examples	O
,	O
which	O
were	O
selected	O
by	O
the	O
max	O
-	O
pooling	O
of	O
the	O
CNNs	O
.	O
For	O
instance	O
,	O
Figure	O
2	O
,	O
corresponding	O
to	O
a	O
feature	O
of	O
filter	O
size	O
2	O
,	O
shows	O
bi	O
-	O
grams	O
(	O
e.g.	O
,	O
"	O
love	O
love	O
"	O
,	O
"	O
love	O
my	O
"	O
,	O
"	O
loves	O
his	O
"	O
,	O
etc	O
.	O
)	O
whose	O
font	O
size	O
corresponds	O
to	O
the	O
feature	O
values	O
of	O
the	O
bi	O
-	O
grams	O
.	O
This	O
is	O
similar	O
to	O
how	O
previous	O
works	O
analyze	O
CNN	O
features	O
(	O
Jacovi	O
et	O
al	O
.	O
,	O
2018;Lertvittayakumjorn	O
and	O
Toni	O
,	O
2019	O
)	O
,	O
and	O
it	O
is	O
equivalent	O
to	O
back	O
-	O
propagating	O
the	O
feature	O
values	O
to	O
the	O
input	O
using	O
LRP	O
and	O
cropping	O
the	O
consecutive	O
input	O
words	O
with	O
non	O
-	O
zero	O
LRP	O
scores	O
to	O
show	O
in	O
the	O
word	O
clouds	O
.	O

Disabling	O
Features	O

As	O
explained	O
earlier	O
,	O
we	O
want	O
to	O
know	O
whether	O
the	O
learned	O
features	O
are	O
valid	O
and	O
relevant	O
to	O
the	O
classification	O
task	O
and	O
whether	O
or	O
not	O
they	O
get	O
appropriate	O
weights	O
from	O
the	O
next	O
layer	O
.	O
This	O
is	O
possible	O
by	O
letting	O
humans	O
consider	O
the	O
word	O
cloud(s	O
)	O
of	O
each	O
feature	O
and	O
tell	O
us	O
which	O
class	O
the	O
feature	O
is	O
relevant	O
to	O
.	O
A	O
word	O
cloud	O
receiving	O
human	O
answers	O
that	O
are	O
different	O
from	O
the	O
class	O
it	O
should	O
support	O
(	O
as	O
indicated	O
by	O
W	O
)	O
exhibits	O
a	O
flaw	O
in	O
the	O
model	O
.	O
For	O
example	O
,	O
if	O
the	O
word	O
cloud	O
in	O
Figure	O
2	O
represents	O
the	O
feature	O
f	O
i	O
in	O
a	O
sentiment	O
analysis	O
task	O
but	O
the	O
i	O
th	O
column	O
of	O
W	O
implies	O
that	O
f	O
i	O
supports	O
the	O
negative	O
sentiment	O
class	O
,	O
we	O
know	O
the	O
model	O
is	O
not	O
correct	O
here	O
.	O
If	O
this	O
word	O
cloud	O
appears	O
in	O
a	O
product	O
categorization	O
task	O
,	O
this	O
is	O
also	O
problematic	O
because	O
the	O
phrases	O
in	O
the	O
word	O
cloud	O
are	O
not	O
discriminative	O
of	O
any	O
product	O
category	O
.	O
Hence	O
,	O
we	O
provide	O
options	O
for	O
the	O
users	O
to	O
disable	O
the	O
features	O
which	O
correspond	O
to	O
any	O
problematic	O
word	O
clouds	O
so	O
that	O
the	O
features	O
do	O
not	O
play	O
a	O
role	O
in	O
the	O
classification	O
.	O
To	O
enable	O
this	O
to	O
happen	O
,	O
we	O
modify	O
M	O
c	O
to	O
be	O
M	O
c	O
where	O
p	O
=	O
M	O
c	O
(	O
f	O
)	O
=	O
softmax((W	O
Q)f	O
+	O
b	O
)	O
and	O
Q	O
∈	O
R	O
|C|×d	O
is	O
a	O
masking	O
matrix	O
with	O
being	O
an	O
element	O
-	O
wise	O
multiplication	O
operator	O
.	O
Initially	O
,	O
all	O
elements	O
in	O
Q	O
are	O
ones	O
which	O
enable	O
all	O
the	O
connections	O
between	O
the	O
features	O
and	O
the	O
output	O
.	O
To	O
disable	O
feature	O
f	O
i	O
,	O
we	O
set	O
the	O
i	O
th	O
column	O
of	O
Q	O
to	O
be	O
a	O
zero	O
vector	O
.	O
After	O
disabling	O
features	O
,	O
we	O
then	O
freeze	O
the	O
parameters	O
of	O
M	O
f	O
and	O
fine	O
-	O
tune	O
the	O
parameters	O
of	O
M	O
c	O
(	O
except	O
the	O
masking	O
matrix	O
Q	O
)	O
with	O
the	O
original	O
training	O
dataset	O
D	O
in	O
the	O
final	O
step	O
.	O

Experimental	O
Setup	O

All	O
datasets	O
and	O
their	O
splits	O
used	O
in	O
the	O
experiments	O
are	O
listed	O
in	O
Table	O
1	O
.	O
We	O
will	O
explain	O
each	O
of	O
them	O
in	O
the	O
following	O
sections	O
.	O
For	O
each	O
classification	O
task	O
,	O
we	O
ran	O
and	O
improved	O
three	O
models	O
,	O
using	O
different	O
random	O
seeds	O
,	O
independently	O
of	O
one	O
another	O
,	O
and	O
the	O
reported	O
results	O
are	O
the	O
average	O
of	O
the	O
three	O
runs	O
.	O
Regarding	O
the	O
models	O
,	O
we	O
used	O
1D	O
CNNs	O
with	O
the	O
same	O
structures	O
for	O
all	O
the	O
tasks	O
and	O
datasets	O
.	O
The	O
convolution	O
layer	O
had	O
three	O
filter	O
sizes	O
[	O
2	O
,	O
3	O
,	O
4	O
]	O
with	O
10	O
filters	O
for	O
each	O
size	O
(	O
i.e.	O
,	O
d	O
=	O
10	O
×	O
3	O
=	O
30	O
)	O
.	O
All	O
the	O
activation	O
functions	O
were	O
ReLU	O
except	O
the	O
softmax	O
at	O
the	O
output	O
layer	O
.	O
The	O
input	O
documents	O
were	O
padded	O
or	O
trimmed	O
to	O
have	O
150	O
words	O
(	O
L	O
=	O
150	O
)	O
.	O
We	O
used	O
pre	O
-	O
trained	O
300	O
-	O
dim	O
GloVe	O
vectors	O
(	O
Pennington	O
et	O
al	O
.	O
,	O
2014	O
)	O
as	O
non	O
-	O
trainable	O
weights	O
in	O
the	O
embedding	O
layers	O
.	O
All	O
the	O
models	O
were	O
implemented	O
using	O
Keras	O
and	O
trained	O
with	O
Adam	O
optimizer	O
.	O
We	O
used	O
iNNvestigate	O
(	O
Alber	O
et	O
al	O
.	O
,	O
2018	O
)	O
to	O
run	O
LRP	O
on	O
CNN	O
features	O
.	O
In	O
particular	O
,	O
we	O
used	O
the	O
LRP	O
-	O
propagation	O
rule	O
to	O
stabilize	O
the	O
relevance	O
scores	O
(	O
=	O
10	O
−7	O
)	O
.	O
Finally	O
,	O
we	O
used	O
Amazon	O
Mechanical	O
Turk	O
(	O
MTurk	O
)	O
to	O
collect	O
crowdsourced	O
responses	O
for	O
selecting	O
features	O
to	O
disable	O
.	O
Each	O
question	O
was	O
answered	O
by	O
ten	O
workers	O
and	O
the	O
answers	O
were	O
aggregated	O
using	O
majority	O
votes	O
or	O
average	O
scores	O
depending	O
on	O
the	O
question	O
type	O
(	O
as	O
explained	O
next	O
)	O
.	O

Exp	O
1	O
:	O
Feasibility	O
Study	O

In	O
this	O
feasibility	O
study	O
,	O
we	O
assessed	O
the	O
effectiveness	O
of	O
word	O
clouds	O
as	O
visual	O
explanations	O
to	O
reveal	O
the	O
behavior	O
of	O
CNN	O
features	O
.	O
We	O
trained	O
CNN	O
models	O
using	O
small	O
training	O
datasets	O
and	O
evaluated	O
the	O
quality	O
of	O
CNN	O
features	O
based	O
on	O
responses	O
from	O
MTurk	O
workers	O
to	O
the	O
feature	O
word	O
clouds	O
.	O

Then	O
we	O
disabled	O
features	O
based	O
on	O
their	O
average	O
quality	O
scores	O
.	O
The	O
assumption	O
was	O
:	O
if	O
the	O
scores	O
of	O
the	O
disabled	O
features	O
correlated	O
with	O
the	O
drop	O
in	O
the	O
model	O
predictive	O
performance	O
,	O
it	O
meant	O
that	O
humans	O
could	O
understand	O
and	O
accurately	O
assess	O
CNN	O
features	O
using	O
word	O
clouds	O
.	O
We	O
used	O
small	O
training	O
datasets	O
so	O
that	O
the	O
trained	O
CNNs	O
had	O
features	O
with	O
different	O
levels	O
of	O
quality	O
.	O
Some	O
features	O
detected	O
useful	O
patterns	O
,	O
while	O
others	O
overfitted	O
the	O
training	O
data	O
.	O

Datasets	O

We	O
used	O
subsets	O
of	O
two	O
datasets	O
:	O
(	O
1	O
)	O
Yelp	O
-predicting	O
sentiments	O
of	O
restaurant	O
reviews	O
(	O
positive	O
or	O
negative	O
)	O
and	O
(	O
2	O
)	O
Amazon	O
Products	O
-classifying	O
product	O
reviews	O
into	O
one	O
of	O
four	O
categories	O
(	O
Clothing	O
Shoes	O
and	O
Jewelry	O
,	O
Digital	O
Music	O
,	O
Office	O
Products	O
,	O
or	O
Toys	O
and	O
Games	O
)	O
(	O
He	O
and	O
McAuley	O
,	O
2016	O
)	O
.	O
We	O
sampled	O
500	O
and	O
100	O
examples	O
to	O
be	O
the	O
training	O
data	O
for	O
Yelp	O
and	O
Amazon	O
Products	O
,	O
respectively	O
.	O

Human	O
Feedback	O
Collection	O
and	O
Usage	O

We	O
used	O
human	O
responses	O
on	O
MTurk	O
to	O
assign	O
ranks	O
to	O
features	O
.	O
As	O
each	O
classifier	O
had	O
30	O
original	O
features	O
(	O
d	O
=	O
30	O
)	O
,	O
we	O
divided	O
them	O
into	O
three	O
ranks	O
(	O
A	O
,	O
B	O
,	O
and	O
C	O
)	O
each	O
of	O
which	O
with	O
10	O
features	O
.	O
We	O
expected	O
that	O
features	O
in	O
rank	O
A	O
are	O
most	O
relevant	O
and	O
useful	O
for	O
the	O
prediction	O
task	O
,	O
and	O
features	O
in	O
rank	O
C	O
least	O
relevant	O
,	O
potentially	O
undermining	O
the	O
performance	O
of	O
the	O
model	O
.	O
To	O
make	O
the	O
annotation	O
more	O
accessible	O
to	O
lay	O
users	O
,	O
we	O
designed	O
the	O
questions	O
to	O
ask	O
whether	O
a	O
given	O
word	O
cloud	O
is	O
(	O
mostly	O
or	O
partially	O
)	O
relevant	O
to	O
one	O
of	O
the	O
classes	O
or	O
not	O
,	O
as	O
shown	O
in	O
Figure	O
3	O
.	O
If	O
the	O
answer	O
matches	O
how	O
the	O
model	O
really	O
uses	O
this	O
feature	O
(	O
as	O
indicated	O
by	O
W	O
)	O
,	O
the	O
feature	O
gets	O
a	O
positive	O
score	O
from	O
this	O
human	O
response	O
.	O
For	O
example	O
,	O
if	O
the	O
CNN	O
feature	O
of	O
the	O
word	O
cloud	O
in	O
Figure	O
3	O
is	O
used	O
by	O
the	O
model	O
for	O
the	O
negative	O
sentiment	O
class	O
,	O
the	O
scores	O
of	O
the	O
five	O
options	O
in	O
the	O
figure	O
are	O
-2	O
,	O
-1	O
,	O
0	O
,	O
1	O
,	O
2	O
,	O
respectively	O
.	O
We	O
collected	O
ten	O
responses	O
for	O
each	O
question	O
and	O
used	O
the	O
average	O
score	O
to	O
sort	O
the	O
features	O
descendingly	O
.	O
After	O
sorting	O
,	O
the	O
1	O
st	O
-10	O
th	O
features	O
,	O
11	O
th	O
-20	O
th	O
features	O
,	O
and	O
21	O
st	O
-30	O
th	O
features	O
are	O
considered	O
as	O
rank	O
A	O
,	O
B	O
,	O
and	O
C	O
,	O
respectively	O
.	O
3	O
To	O
show	O
the	O
effects	O
of	O
feature	O
disabling	O
,	O
we	O
compared	O
the	O
original	O
model	O
M	O
with	O
the	O
modified	O
model	O
M	O
with	O
features	O
in	O
rank	O
X	O
disabled	O
where	O
X	O
∈	O
{	O
A	O
,	O
B	O
,	O
C	O
,	O
A	O
and	O
B	O
,	O
A	O
and	O
C	O
,	O
B	O
and	O
C	O
}	O
.	O

Results	O
and	O
Discussions	O

Figure	O
4	O
shows	O
the	O
distribution	O
of	O
average	O
feature	O
scores	O
from	O
one	O
of	O
the	O
three	O
CNN	O
instances	O
for	O
the	O
Yelp	O
dataset	O
.	O
Examples	O
of	O
the	O
word	O
clouds	O
from	O
each	O
rank	O
are	O
displayed	O
in	O
Figure	O
5	O
.	O
We	O
can	O
clearly	O
see	O
dissimilar	O
qualities	O
of	O
the	O
three	O
features	O
.	O
Some	O
participants	O
answered	O
that	O
the	O
rank	O
B	O
feature	O
in	O
Figure	O
5	O
was	O
relevant	O
to	O
the	O
positive	O
class	O
(	O
probably	O
due	O
to	O
the	O
word	O
'	O
delicious	O
'	O
)	O
,	O
and	O
the	O
weights	O
of	O
this	O
feature	O
in	O
W	O
agreed	O
(	O
Positive	O
:	O
Negative	O
=	O
0.137:-0.135	O
)	O
.	O
Interestingly	O
,	O
the	O
rank	O
C	O
feature	O
in	O
Figure	O
5	O
got	O
a	O
negative	O
score	O
because	O
some	O
participants	O
believed	O
that	O
this	O
word	O
cloud	O
was	O
relevant	O
to	O
the	O
positive	O
class	O
,	O
but	O
actually	O
the	O
model	O
used	O
this	O
feature	O
as	O
evidence	O
for	O
the	O
negative	O
class	O
(	O
Positive	O
:	O
Negative	O
=	O
0.209:0.385	O
)	O
.	O

Considering	O
all	O
the	O
three	O
runs	O
,	O
Figure	O
6	O
(	O
top	O
)	O
shows	O
the	O
average	O
macro	O
F1	O
score	O
of	O
the	O
original	O
model	O
(	O
the	O
blue	O
line	O
)	O
and	O
of	O
each	O
modified	O
model	O
.	O
The	O
order	O
of	O
the	O
performance	O
drops	O
is	O
AB	O
>	O
A	O
>	O
AC	O
>	O
BC	O
>	O
B	O
>	O
Original	O
>	O
C.	O
This	O
makes	O
sense	O
because	O
disabling	O
important	O
features	O
(	O
rank	O
A	O
and/or	O
B	O
)	O
caused	O
larger	O
performance	O
drops	O
,	O
and	O
the	O
overall	O
results	O
are	O
consistent	O
with	O
the	O
average	O
fea-	O
ture	O
scores	O
given	O
by	O
the	O
participants	O
(	O
as	O
in	O
Figure	O
4	O
)	O
.	O
It	O
confirms	O
that	O
using	O
word	O
clouds	O
is	O
an	O
effective	O
way	O
to	O
assess	O
CNN	O
features	O
.	O
Also	O
,	O
it	O
is	O
worth	O
noting	O
that	O
the	O
macro	O
F1	O
of	O
the	O
model	O
slightly	O
increased	O
when	O
we	O
disabled	O
the	O
low	O
-	O
quality	O
features	O
(	O
rank	O
C	O
)	O
.	O
This	O
shows	O
that	O
humans	O
can	O
improve	O
the	O
model	O
by	O
disabling	O
irrelevant	O
features	O
.	O

The	O
CNNs	O
for	O
the	O
Amazon	O
Products	O
dataset	O
also	O
behaved	O
in	O
a	O
similar	O
way	O
(	O
Figure	O
6	O
-bottom	O
)	O
,	O
except	O
that	O
disabling	O
rank	O
C	O
features	O
slightly	O
undermined	O
,	O
not	O
increased	O
,	O
performance	O
.	O
This	O
implies	O
that	O
even	O
the	O
rank	O
C	O
features	O
contain	O
a	O
certain	O
amount	O
of	O
useful	O
knowledge	O
for	O
this	O
classifier	O
.	O
4	O

6	O
Exp	O
2	O
:	O
Training	O
Data	O
with	O
Biases	O

Given	O
a	O
biased	O
training	O
dataset	O
,	O
a	O
text	O
classifier	O
may	O
absorb	O
the	O
biases	O
and	O
produce	O
biased	O
predictions	O
against	O
some	O
sub	O
-	O
populations	O
.	O
We	O
hypothesize	O
that	O
if	O
the	O
biases	O
are	O
captured	O
by	O
some	O
of	O
the	O
learned	O
features	O
,	O
we	O
can	O
apply	O
FIND	O
to	O
disable	O
such	O
features	O
and	O
reduce	O
the	O
model	O
biases	O
.	O

Datasets	O
and	O
Metrics	O

We	O
focus	O
on	O
reducing	O
gender	O
bias	O
of	O
CNN	O
models	O
trained	O
on	O
two	O
datasets	O
-Biosbias	O
(	O
De	O
-	O
Arteaga	O
et	O
al	O
.	O
,	O
2019	O
)	O
and	O
Waseem	O
(	O
Waseem	O
and	O
Hovy	O
,	O
2016	O
)	O
.	O
For	O
Biosbias	O
,	O
the	O
task	O
is	O
predicting	O
the	O
occupation	O
of	O
a	O
given	O
bio	O
paragraph	O
,	O
i.e.	O
,	O
whether	O
the	O
person	O
is	O
'	O
a	O
surgeon	O
'	O
(	O
class	O
0	O
)	O
or	O
'	O
a	O
nurse	O
'	O
(	O
class	O
1	O
)	O
.	O
Due	O
to	O
the	O
gender	O
imbalance	O
in	O
each	O
occupation	O
,	O
a	O
classifier	O
usually	O
exploits	O
gender	O
information	O
when	O
making	O
predictions	O
.	O
As	O
a	O
result	O
,	O
bios	O
of	O
female	O
surgeons	O
and	O
male	O
nurses	O
are	O
often	O
misclassified	O
.	O
For	O
Waseem	O
,	O
the	O
task	O
is	O
abusive	O
language	O
detection	O
-assessing	O
if	O
a	O
given	O
text	O
is	O
abusive	O
(	O
class	O
1	O
)	O
or	O
not	O
abusive	O
(	O
class	O
0	O
)	O
.	O
Previous	O
work	O
found	O
that	O
this	O
dataset	O
contains	O
a	O
strong	O
negative	O
bias	O
against	O
females	O
(	O
Park	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O
In	O
other	O
words	O
,	O
texts	O
related	O
to	O
females	O
are	O
usually	O
classified	O
as	O
abusive	O
although	O
the	O
texts	O
themselves	O
are	O
not	O
abusive	O
at	O
all	O
.	O
Also	O
,	O
we	O
tested	O
the	O
models	O
,	O
trained	O
on	O
the	O
Waseem	O
dataset	O
,	O
using	O
another	O
abusive	O
language	O
detection	O
dataset	O
,	O
Wikitoxic	O
(	O
Thain	O
et	O
al	O
.	O
,	O
2017	O
)	O
,	O
to	O
assess	O
generalizability	O
of	O
the	O
models	O
.	O
To	O
quantify	O
gender	O
biases	O
,	O
we	O
adopted	O
two	O
metrics	O
-false	O
positive	O
equality	O
difference	O
(	O
FPED	O
)	O
and	O
false	O
negative	O
equality	O
difference	O
(	O
FNED	O
)	O
(	O
Dixon	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O

The	O
lower	O
these	O
metrics	O
are	O
,	O
the	O
less	O
biases	O
the	O
model	O
has	O
.	O
4	O
We	O
also	O
conducted	O
the	O
same	O
experiments	O
here	O
with	O
bidirectional	O
LSTM	O
networks	O
(	O
BiLSTMs	O
)	O
which	O
required	O
a	O
different	O
way	O
to	O
generate	O
the	O
word	O
clouds	O
(	O
see	O
Appendix	O
C	O
)	O
.	O
The	O
results	O
on	O
BiLSTMs	O
,	O
however	O
,	O
are	O
not	O
as	O
promising	O
as	O
on	O
CNNs	O
.	O
This	O
might	O
be	O
because	O
the	O
way	O
we	O
created	O
word	O
clouds	O
for	O
each	O
BiLSTM	O
feature	O
was	O
not	O
an	O
accurate	O
way	O
to	O
reveal	O
its	O
behavior	O
.	O
Unlike	O
for	O
CNNs	O
,	O
understanding	O
recurrent	O
neural	O
network	O
features	O
for	O
text	O
classification	O
is	O
still	O
an	O
open	O
problem	O
.	O

Human	O
Feedback	O
Collection	O
and	O
Usage	O

Unlike	O
the	O
interface	O
in	O
Figure	O
3	O
,	O
for	O
each	O
word	O
cloud	O
,	O
we	O
asked	O
the	O
participants	O
to	O
select	O
the	O
relevant	O
class	O
from	O
three	O
options	O
(	O
Biosbias	O
:	O
surgeon	O
,	O
nurse	O
,	O
it	O
could	O
be	O
either	O
/	O
Waseem	O
:	O
abusive	O
,	O
nonabusive	O
,	O
it	O
could	O
be	O
either	O
)	O
.	O
The	O
feature	O
will	O
be	O
disabled	O
if	O
the	O
majority	O
vote	O
does	O
not	O
select	O
the	O
class	O
suggested	O
by	O
the	O
weight	O
matrix	O
W.	O
To	O
ensure	O
that	O
the	O
participants	O
do	O
not	O
use	O
their	O
biases	O
while	O
answering	O
our	O
questions	O
,	O
we	O
firmly	O
mentioned	O
in	O
the	O
instructions	O
that	O
gender	O
-	O
related	O
terms	O
should	O
not	O
be	O
used	O
as	O
an	O
indicator	O
for	O
one	O
or	O
the	O
other	O
class	O
.	O

Results	O
and	O
Discussions	O

The	O
results	O
of	O
this	O
experiment	O
are	O
displayed	O
in	O
Figure	O
7	O
.	O
For	O
Biosbias	O
,	O
on	O
average	O
,	O
the	O
participants	O
'	O
responses	O
suggested	O
us	O
to	O
disable	O
11.33	O
out	O
of	O
30	O
CNN	O
features	O
.	O
By	O
doing	O
so	O
,	O
the	O
FPED	O
of	O
the	O
models	O
decreased	O
from	O
0.250	O
to	O
0.163	O
,	O
and	O
the	O
FNED	O
decreased	O
from	O
0.338	O
to	O
0.149	O
.	O
After	O
investigating	O
the	O
word	O
clouds	O
of	O
the	O
CNN	O
features	O
,	O
we	O
found	O
that	O
some	O
of	O
them	O
detected	O
patterns	O
containing	O
both	O
gender	O
-	O
related	O
terms	O
and	O
occupation	O
-	O
related	O
terms	O
such	O
as	O
"	O
his	O
surgical	O
expertise	O
"	O
and	O
"	O
she	O
supervises	O
nursing	O
students	O
"	O
.	O
Most	O
of	O
the	O
MTurk	O
participants	O
answered	O
that	O
these	O
word	O
clouds	O
were	O
relevant	O
to	O
the	O
occupations	O
,	O
and	O
thus	O
the	O
corresponding	O
features	O
were	O
not	O
disabled	O
.	O
However	O
,	O
we	O
believe	O
that	O
these	O
features	O
might	O
contain	O
gender	O
biases	O
.	O
So	O
,	O
we	O
asked	O
one	O
annotator	O
to	O
consider	O
all	O
the	O
word	O
clouds	O
again	O
and	O
disable	O
every	O
feature	O
for	O
which	O
the	O
prominent	O
n	O
-	O
gram	O
patterns	O
contained	O
any	O
genderrelated	O
terms	O
,	O
no	O
matter	O
whether	O
the	O
patterns	O
detect	O
occupation	O
-	O
related	O
terms	O
.	O
With	O
this	O
new	O
disabling	O
policy	O
,	O
12	O
out	O
of	O
30	O
features	O
were	O
disabled	O
on	O
average	O
,	O
and	O
the	O
model	O
biases	O
further	O
decreased	O
,	O
as	O
shown	O
in	O
Figure	O
7	O
(	O
Debugged	O
(	O
One	O
)	O
)	O
.	O
The	O
sideeffect	O
of	O
disabling	O
33	O
%	O
of	O
all	O
the	O
features	O
here	O
was	O
only	O
a	O
slight	O
drop	O
in	O
the	O
macro	O
F1	O
from	O
0.950	O
to	O
0.933	O
.	O
Hence	O
,	O
our	O
framework	O
was	O
successful	O
in	O
reducing	O
gender	O
biases	O
without	O
severe	O
negative	O
effects	O
in	O
classification	O
performance	O
.	O

Concerning	O
the	O
abusive	O
language	O
detection	O
task	O
,	O
on	O
average	O
,	O
the	O
MTurk	O
participants	O
'	O
responses	O
suggested	O
us	O
to	O
disable	O
12	O
out	O
of	O
30	O
CNN	O
features	O
.	O
Unlike	O
Biosbias	O
,	O
disabling	O
features	O
based	O
on	O
MTurk	O
responses	O
unexpectedly	O
increased	O
the	O
gender	O
bias	O
for	O
both	O
Waseem	O
and	O
Wikitoxic	O
datasets	O
.	O
However	O
,	O
we	O
found	O
one	O
similar	O
finding	O
to	O
Biosbias	O
,	O
that	O
many	O
of	O
the	O
CNN	O
features	O
captured	O
n	O
-	O
grams	O
which	O
were	O
both	O
abusive	O
and	O
related	O
to	O
a	O
gender	O
such	O
as	O
'	O
these	O
girls	O
are	O
terrible	O
'	O
and	O
'	O
of	O
raping	O
slave	O
girls	O
'	O
,	O
and	O
these	O
features	O
were	O
not	O
yet	O
disabled	O
.	O
So	O
,	O
we	O
asked	O
one	O
annotator	O
to	O
disable	O
the	O
features	O
using	O
the	O
new	O
"	O
brutal	O
"	O
policy	O
-disabling	O
all	O
which	O
involved	O
gender	O
words	O
even	O
though	O
some	O
of	O
them	O
also	O
detected	O
abusive	O
words	O
.	O
By	O
disabling	O
18	O
out	O
of	O
30	O
features	O
on	O
average	O
,	O
the	O
gender	O
biases	O
were	O
reduced	O
for	O
both	O
datasets	O
(	O
except	O
FPED	O
on	O
Wikitoxic	O
which	O
stayed	O
close	O
to	O
the	O
original	O
value	O
)	O
.	O
Another	O
consequence	O
was	O
that	O
we	O
sacrificed	O
4	O
%	O
and	O
1	O
%	O
macro	O
F1	O
on	O
the	O
Waseem	O
and	O
Wikitoxic	O
datasets	O
,	O
respectively	O
.	O
This	O
finding	O
is	O
consistent	O
with	O
(	O
Park	O
et	O
al	O
.	O
,	O
2018	O
)	O
that	O
reducing	O
the	O
bias	O
and	O
maintaining	O
the	O
classification	O
performance	O
at	O
the	O
same	O
time	O
is	O
very	O
challenging	O
.	O

Exp	O
3	O
:	O
Dataset	O
Shift	O

Dataset	O
shift	O
is	O
a	O
problem	O
where	O
the	O
joint	O
distribution	O
of	O
inputs	O
and	O
outputs	O
differs	O
between	O
training	O
and	O
test	O
stage	O
(	O
Quionero	O
-	O
Candela	O
et	O
al	O
.	O
,	O
2009	O
)	O
.	O
Many	O
classifiers	O
perform	O
poorly	O
under	O
dataset	O
shift	O
because	O
some	O
of	O
the	O
learned	O
features	O
are	O
inapplicable	O
(	O
or	O
sometimes	O
even	O
harmful	O
)	O
to	O
classify	O
test	O
documents	O
.	O
We	O
hypothesize	O
that	O
FIND	O
is	O
useful	O
for	O
investigating	O
the	O
learned	O
features	O
and	O
disabling	O
the	O
overfitting	O
ones	O
to	O
increase	O
the	O
generalizability	O
of	O
the	O
model	O
.	O

Datasets	O

We	O
considered	O
two	O
tasks	O
in	O
this	O
experiment	O
.	O
The	O
first	O
task	O
aims	O
to	O
classify	O
"	O
Christianity	O
"	O
vs	O
"	O
Atheism	O
"	O
documents	O
from	O
the	O
20	O
Newsgroups	O
dataset	O
5	O
.	O
This	O
dataset	O
is	O
special	O
because	O
it	O
contains	O
a	O
lot	O
of	O
artifacts	O
-tokens	O
(	O
e.g.	O
,	O
person	O
names	O
,	O
punctuation	O
marks	O
)	O
which	O
are	O
not	O
relevant	O
,	O
but	O
strongly	O
co	O
-	O
occur	O
with	O
one	O
of	O
the	O
classes	O
.	O
For	O
evaluation	O
,	O
we	O
used	O
the	O
Religion	O
dataset	O
by	O
Ribeiro	O
et	O
al	O
.	O
(	O
2016	O
)	O
,	O
containing	O
"	O
Christianity	O
"	O
and	O
"	O
Atheism	O
"	O
web	O
pages	O
,	O
as	O
a	O
target	O
dataset	O
.	O
The	O
second	O
task	O
is	O
sentiment	O
analysis	O
.	O
We	O
used	O
,	O
as	O
a	O
training	O
dataset	O
,	O
Amazon	O
Clothes	O
,	O
with	O
reviews	O
of	O
clothing	O
,	O
shoes	O
,	O
and	O
jewelry	O
products	O
(	O
He	O
and	O
McAuley	O
,	O
2016	O
)	O
,	O
and	O
as	O
test	O
sets	O
three	O
out	O
-	O
of	O
-	O
distribution	O
datasets	O
-Amazon	O
Music	O
(	O
He	O
and	O
McAuley	O
,	O
2016	O
)	O
,	O
Amazon	O
Mixed	O
,	O
and	O
the	O
Yelp	O
dataset	O
(	O
which	O
was	O
used	O
in	O
Experiment	O
1	O
)	O
.	O
Amazon	O
Music	O
contains	O
only	O
reviews	O
from	O
the	O
"	O
Digital	O
Music	O
"	O
product	O
category	O
which	O
was	O
found	O
to	O
have	O
an	O
extreme	O
distribution	O
shift	O
from	O
the	O
clothes	O
category	O
(	O
Hendrycks	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
Amazon	O
Mixed	O
compiles	O
the	O
reviews	O
from	O
various	O
kinds	O
of	O
products	O
,	O
while	O
Yelp	O
focuses	O
on	O
restaurant	O
reviews	O
.	O

Human	O
Feedback	O
Collection	O
and	O
Usage	O

We	O
collected	O
responses	O
from	O
MTurk	O
workers	O
using	O
the	O
same	O
user	O
interfaces	O
as	O
in	O
Experiment	O
2	O
.	O
Simply	O
put	O
,	O
we	O
asked	O
the	O
workers	O
to	O
select	O
a	O
class	O
which	O
was	O
relevant	O
to	O
a	O
given	O
word	O
cloud	O
and	O
checked	O
if	O
the	O
majority	O
vote	O
agreed	O
with	O
the	O
weights	O
in	O
W.	O

Results	O
and	O
Discussions	O

For	O
the	O
first	O
task	O
,	O
on	O
average	O
,	O
14.33	O
out	O
of	O
30	O
features	O
were	O
disabled	O
and	O
the	O
macro	O
F1	O
scores	O
of	O
the	O
20Newsgroups	O
before	O
and	O
after	O
debugging	O
are	O
0.853	O
and	O
0.828	O
,	O
respectively	O
.	O
The	O
same	O
metrics	O
of	O
the	O
Religion	O
dataset	O
are	O
0.731	O
and	O
0.799	O
.	O
This	O
shows	O
that	O
disabling	O
irrelevant	O
features	O
mildly	O
undermined	O
the	O
predictive	O
performance	O
on	O
the	O
indistribution	O
dataset	O
,	O
but	O
clearly	O
enhanced	O
the	O
performance	O
on	O
the	O
out	O
-	O
of	O
-	O
distribution	O
dataset	O
(	O
see	O
Figure	O
8	O
,	O
left	O
)	O
.	O
This	O
is	O
especially	O
evident	O
for	O
the	O
Atheism	O
class	O
for	O
which	O
the	O
F1	O
score	O
increased	O
around	O
15	O
%	O
absolute	O
.	O
We	O
noticed	O
from	O
the	O
word	O
clouds	O
that	O
many	O
prominent	O
words	O
for	O
the	O
Atheism	O
class	O
learned	O
by	O
the	O
models	O
are	O
person	O
names	O
(	O
e.g.	O
,	O
Keith	O
,	O
Gregg	O
,	O
Schneider	O
)	O
and	O
these	O
are	O
not	O
applicable	O
to	O
the	O
Religion	O
dataset	O
.	O
Forcing	O
the	O
models	O
to	O
use	O
only	O
relevant	O
features	O
(	O
detecting	O
terms	O
like	O
'	O
atheists	O
'	O
and	O
'	O
science	O
'	O
)	O
,	O
therefore	O
,	O
increased	O
the	O
macro	O
F1	O
on	O
the	O
Religion	O
dataset	O
.	O

Unlike	O
20Newsgroups	O
,	O
Amazon	O
Clothes	O
does	O
not	O
seem	O
to	O
have	O
obvious	O
artifacts	O
.	O
Still	O
,	O
the	O
re	O
-	O
sponses	O
from	O
crowd	O
workers	O
suggested	O
that	O
we	O
disable	O
6	O
features	O
.	O
The	O
disabled	O
features	O
were	O
correlated	O
to	O
,	O
but	O
not	O
the	O
reason	O
for	O
,	O
the	O
associated	O
class	O
.	O
For	O
instance	O
,	O
one	O
of	O
the	O
disabled	O
features	O
was	O
highly	O
activated	O
by	O
the	O
pattern	O
"	O
my	O
....	O
year	O
old	O
"	O
which	O
often	O
appeared	O
in	O
positive	O
reviews	O
such	O
as	O
"	O
my	O
3	O
year	O
old	O
son	O
loves	O
this	O
.	O
"	O
.	O
However	O
,	O
these	O
correlated	O
features	O
are	O
not	O
very	O
useful	O
for	O
the	O
three	O
outof	O
-	O
distribution	O
datasets	O
(	O
Music	O
,	O
Mixed	O
,	O
and	O
Yelp	O
)	O
.	O
Disabling	O
them	O
made	O
the	O
model	O
focus	O
more	O
on	O
the	O
right	O
evidence	O
and	O
increased	O
the	O
average	O
macro	O
F1	O
for	O
the	O
three	O
datasets	O
,	O
as	O
shown	O
in	O
Figure	O
8	O
(	O
right	O
)	O
.	O
Nonetheless	O
,	O
the	O
performance	O
improvement	O
here	O
was	O
not	O
as	O
apparent	O
as	O
in	O
the	O
previous	O
task	O
because	O
,	O
even	O
without	O
feature	O
disabling	O
,	O
the	O
majority	O
of	O
the	O
features	O
are	O
relevant	O
to	O
the	O
task	O
and	O
can	O
lead	O
the	O
model	O
to	O
the	O
correct	O
predictions	O
in	O
most	O
cases	O
.	O
6	O

Discussion	O
and	O
Conclusions	O

We	O
proposed	O
FIND	O
,	O
a	O
framework	O
which	O
enables	O
humans	O
to	O
debug	O
deep	O
text	O
classifiers	O
by	O
disabling	O
irrelevant	O
or	O
harmful	O
features	O
.	O
Using	O
the	O
proposed	O
framework	O
on	O
CNN	O
text	O
classifiers	O
,	O
we	O
found	O
that	O
(	O
i	O
)	O
word	O
clouds	O
generated	O
by	O
running	O
LRP	O
on	O
the	O
training	O
data	O
accurately	O
revealed	O
the	O
behaviors	O
of	O
CNN	O
features	O
,	O
(	O
ii	O
)	O
some	O
of	O
the	O
learned	O
features	O
might	O
be	O
more	O
useful	O
to	O
the	O
task	O
than	O
the	O
others	O
and	O
(	O
iii	O
)	O
disabling	O
the	O
irrelevant	O
or	O
harmful	O
features	O
could	O
improve	O
the	O
model	O
predictive	O
performance	O
and	O
reduce	O
unintended	O
biases	O
in	O
the	O
model	O
.	O

Generalization	O
to	O
Other	O
Models	O

In	O
order	O
to	O
generalize	O
the	O
framework	O
beyond	O
CNNs	O
,	O
there	O
are	O
two	O
questions	O
to	O
consider	O
.	O
First	O
,	O
what	O
is	O
an	O
effective	O
way	O
to	O
understand	O
each	O
feature	O
?	O
We	O
exemplified	O
this	O
with	O
two	O
word	O
clouds	O
representing	O
each	O
BiLSTM	O
feature	O
in	O
Appendix	O
C	O
,	O
and	O
we	O
plan	O
to	O
experiment	O
with	O
advanced	O
visualizations	O
such	O
as	O
LSTMVis	O
(	O
Strobelt	O
et	O
al	O
.	O
,	O
2018	O
)	O
in	O
the	O
future	O
.	O
Second	O
,	O
can	O
we	O
make	O
the	O
model	O
features	O
more	O
interpretable	O
?	O
For	O
example	O
,	O
using	O
ReLU	O
as	O
activation	O
functions	O
in	O
LSTM	O
cells	O
(	O
instead	O
of	O
tanh	O
)	O
renders	O
the	O
features	O
non	O
-	O
negative	O
.	O
So	O
,	O
they	O
can	O
be	O
summarized	O
using	O
one	O
word	O
cloud	O
which	O
is	O
more	O
practical	O
for	O
debugging	O
.	O

In	O
general	O
,	O
the	O
principle	O
of	O
FIND	O
is	O
understanding	O
the	O
features	O
and	O
then	O
disabling	O
the	O
irrelevant	O
ones	O
.	O
The	O
process	O
makes	O
visualizations	O
and	O
interpretability	O
more	O
actionable	O
.	O
Over	O
the	O
past	O
few	O
years	O
,	O
we	O
have	O
seen	O
rapid	O
growth	O
of	O
scientific	O
research	O
in	O
both	O
topics	O
(	O
visualizations	O
and	O
interpretability	O
)	O
aiming	O
to	O
understand	O
many	O
emerging	O
advanced	O
models	O
including	O
the	O
popular	O
transformer	O
-	O
based	O
models	O
(	O
Jo	O
and	O
Myaeng	O
,	O
2020;Voita	O
et	O
al	O
.	O
,	O
2019;Hoover	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O
We	O
believe	O
that	O
our	O
work	O
will	O
inspire	O
other	O
researchers	O
to	O
foster	O
advances	O
in	O
both	O
topics	O
towards	O
the	O
more	O
tangible	O
goal	O
of	O
model	O
debugging	O
.	O

Generalization	O
to	O
Other	O
Tasks	O

FIND	O
is	O
suitable	O
for	O
any	O
text	O
classification	O
tasks	O
where	O
a	O
model	O
might	O
learn	O
irrelevant	O
or	O
harmful	O
features	O
during	O
training	O
.	O
It	O
is	O
also	O
convenient	O
to	O
use	O
since	O
only	O
the	O
trained	O
model	O
and	O
the	O
training	O
data	O
are	O
required	O
as	O
input	O
.	O
Moreover	O
,	O
it	O
can	O
address	O
many	O
problems	O
simultaneously	O
such	O
as	O
removing	O
religious	O
and	O
racial	O
bias	O
together	O
with	O
gender	O
bias	O
even	O
if	O
we	O
might	O
not	O
be	O
aware	O
of	O
such	O
problems	O
before	O
using	O
FIND	O
.	O
In	O
general	O
cases	O
,	O
FIND	O
is	O
at	O
least	O
useful	O
for	O
model	O
verification	O
.	O

For	O
future	O
work	O
,	O
it	O
would	O
be	O
interesting	O
to	O
extend	O
FIND	O
to	O
other	O
NLP	O
tasks	O
,	O
e.g.	O
,	O
question	O
answering	O
and	O
natural	O
language	O
inference	O
.	O
This	O
will	O
require	O
some	O
modifications	O
to	O
understand	O
how	O
the	O
features	O
capture	O
relationships	O
between	O
two	O
input	O
texts	O
.	O

Limitations	O

Nevertheless	O
,	O
FIND	O
has	O
some	O
limitations	O
.	O
First	O
,	O
the	O
word	O
clouds	O
may	O
reveal	O
sensitive	O
contents	O
in	O
the	O
training	O
data	O
to	O
human	O
debuggers	O
.	O
Second	O
,	O
the	O
more	O
hidden	O
features	O
the	O
model	O
has	O
,	O
the	O
more	O
human	O
effort	O
FIND	O
needs	O
for	O
debugging	O
.	O
For	O
instance	O
,	O
BERT	O
-	O
base	O
(	O
Devlin	O
et	O
al	O
.	O
,	O
2019	O
)	O
has	O
768	O
features	O
(	O
before	O
the	O
final	O
dense	O
layer	O
)	O
which	O
require	O
lots	O
of	O
human	O
effort	O
to	O
perform	O
investigation	O
.	O
In	O
this	O
case	O
,	O
it	O
would	O
be	O
more	O
efficient	O
to	O
use	O
FIND	O
to	O
disable	O
attention	O
heads	O
rather	O
than	O
individual	O
features	O
(	O
Voita	O
et	O
al	O
.	O
,	O
2019	O
)	O
.	O
Third	O
,	O
it	O
is	O
possible	O
that	O
one	O
feature	O
detects	O
several	O
patterns	O
(	O
Jacovi	O
et	O
al	O
.	O
,	O
2018	O
)	O
and	O
it	O
will	O
be	O
difficult	O
to	O
disable	O
the	O
feature	O
if	O
some	O
of	O
the	O
detected	O
patterns	O
are	O
useful	O
while	O
the	O
others	O
are	O
harmful	O
.	O
Hence	O
,	O
FIND	O
would	O
be	O
more	O
effective	O
when	O
used	O
together	O
with	O
disentangled	O
text	O
representations	O
(	O
Cheng	O
et	O
al	O
.	O
,	O
2020	O
)	O
.	O

A	O
Layer	O
-	O
wise	O
Relevance	O
Propagation	O

Layer	O
-	O
wise	O
Relevance	O
Propagation	O
(	O
LRP	O
)	O
is	O
a	O
technique	O
for	O
explaining	O
predictions	O
of	O
neural	O
networks	O
in	O
terms	O
of	O
importance	O
scores	O
of	O
input	O
features	O
(	O
Bach	O
et	O
al	O
.	O
,	O
2015	O
)	O
.	O
Originally	O
,	O
it	O
was	O
devised	O
to	O
explain	O
predictions	O
of	O
image	O
classifiers	O
by	O
creating	O
a	O
heatmap	O
on	O
the	O
input	O
image	O
highlighting	O
pixels	O
that	O
are	O
important	O
for	O
the	O
classification	O
.	O
Then	O
Arras	O
et	O
al	O
.	O
(	O
2016	O
)	O
and	O
Arras	O
et	O
al	O
.	O
(	O
2017	O
)	O
extended	O
LRP	O
to	O
work	O
on	O
CNNs	O
and	O
RNNs	O
for	O
text	O
classification	O
,	O
respectively	O
.	O

Consider	O
a	O
neuron	O
k	O
whose	O
value	O
is	O
computed	O
using	O
n	O
neurons	O
in	O
the	O
previous	O
layer	O
,	O

x	O
k	O
=	O
g	O
(	O
n	O
j=1	O
x	O
j	O
w	O
jk	O
+	O
b	O
k	O
)	O

where	O
x	O
k	O
is	O
the	O
value	O
of	O
the	O
neuron	O
k	O
,	O
g	O
is	O
a	O
nonlinear	O
activation	O
function	O
,	O
w	O
jk	O
and	O
b	O
k	O
are	O
weights	O
and	O
bias	O
in	O
the	O
network	O
,	O
respectively	O
.	O
We	O
can	O
see	O
that	O
the	O
contribution	O
of	O
a	O
single	O
node	O
j	O
to	O
the	O
value	O
of	O
the	O
node	O
k	O
is	O

z	O
jk	O
=	O
x	O
j	O
w	O
jk	O
+	O
b	O
k	O
n	O

assuming	O
that	O
the	O
bias	O
term	O
b	O
k	O
is	O
distributed	O
equally	O
to	O
the	O
n	O
neurons	O
.	O
LRP	O
works	O
by	O
propagating	O
the	O
activation	O
of	O
a	O
neuron	O
of	O
interest	O
back	O
through	O
the	O
previous	O
layers	O
in	O
the	O
network	O
proportionally	O
.	O
We	O
call	O
the	O
value	O
each	O
neuron	O
receives	O
a	O
relevance	O
score	O
(	O
R	O
)	O
of	O
the	O
neuron	O
.	O
To	O
back	O
propagate	O
,	O
if	O
the	O
relevance	O
score	O
of	O
the	O
neuron	O
k	O
is	O
R	O
k	O
,	O
the	O
relevance	O
score	O
that	O
the	O
neuron	O
j	O
receives	O
from	O
the	O
neuron	O

k	O
is	O
R	O
j←k	O
=	O
z	O
jk	O
n	O
j	O
=	O
1	O
z	O
j	O
k	O
R	O
k	O

To	O
make	O
the	O
relevance	O
propagation	O
more	O
stable	O
,	O
we	O
add	O
a	O
small	O
positive	O
number	O
(	O
as	O
a	O
stabilizer	O
)	O
to	O
the	O
denominator	O
of	O
the	O
propagation	O
rule	O
:	O

R	O
j←k	O
=	O
z	O
jk	O
+	O
n	O
j	O
=	O
1	O
z	O
j	O
k	O
R	O
k	O

We	O
used	O
this	O
propagation	O
rule	O
,	O
so	O
called	O
LRP-	O
,	O
in	O
the	O
experiments	O
of	O
this	O
paper	O
.	O
For	O
more	O
details	O
about	O
LRP	O
propagation	O
rules	O
,	O
please	O
see	O
Montavon	O
et	O
al	O
.	O
(	O
2019	O
)	O
.	O

To	O
explain	O
a	O
prediction	O
of	O
a	O
CNN	O
text	O
classifier	O
,	O
we	O
propagate	O
an	O
activation	O
value	O
of	O
the	O
output	O
node	O
back	O
to	O
the	O
word	O
embedding	O
matrix	O
.	O
After	O
that	O
,	O
the	O
relevance	O
score	O
of	O
an	O
input	O
word	O
equals	O
the	O
sum	O
of	O
relevance	O
scores	O
each	O
dimension	O
of	O
its	O
word	O
vector	O
receives	O
.	O
However	O
,	O
in	O
this	O
paper	O
,	O
we	O
want	O
to	O
analyze	O
the	O
hidden	O
features	O
rather	O
than	O
the	O
output	O
,	O
so	O
we	O
start	O
back	O
propagating	O
from	O
the	O
hidden	O
features	O
instead	O
to	O
capture	O
patterns	O
of	O
input	O
words	O
which	O
highly	O
activate	O
the	O
features	O
.	O

B	O
Multiclass	O
Classification	O

As	O
shown	O
in	O
Figure	O
9	O
,	O
we	O
used	O
a	O
slightly	O
different	O
user	O
interface	O
in	O
Experiment	O
1	O
for	O
the	O
Amazon	O
Products	O
dataset	O
which	O
is	O
a	O
multiclass	O
classification	O
task	O
.	O
In	O
this	O
setting	O
,	O
we	O
did	O
not	O
provide	O
the	O
options	O
for	O
mostly	O
and	O
partly	O
relevant	O
;	O
otherwise	O
,	O
there	O
would	O
have	O
been	O
nine	O
options	O
per	O
question	O
which	O
are	O
too	O
many	O
for	O
the	O
participants	O
to	O
answer	O
accurately	O
.	O
With	O
the	O
user	O
interface	O
in	O
Figure	O
9	O
,	O
we	O
gave	O
a	O
score	O
to	O
the	O
feature	O
f	O
i	O
based	O
on	O
the	O
participant	O
answer	O
.	O
To	O
explain	O
,	O
we	O
re	O
-	O
scaled	O
values	O
in	O
the	O
i	O
th	O
column	O
of	O
W	O
to	O
be	O
in	O
the	O
range	O
[	O
0,1	O
]	O
using	O
min	O
-	O
max	O
normalization	O
and	O
gave	O
the	O
normalized	O
value	O
of	O
the	O
chosen	O
class	O
as	O
a	O
score	O
to	O
the	O
feature	O
f	O
i	O
.	O
If	O
the	O
participant	O
selects	O
None	O
,	O
this	O
feature	O
gets	O
a	O
zero	O
score	O
.	O
The	O
distribution	O
of	O
the	O
average	O
feature	O
scores	O
for	O
this	O
task	O
(	O
one	O
CNN	O
)	O
is	O
displayed	O
in	O
Figure	O
10	O
.	O

C	O
Bidirectional	O
LSTM	O
networks	O

To	O
understand	O
BiLSTM	O
features	O
,	O
we	O
created	O
two	O
word	O
clouds	O
for	O
each	O
feature	O
.	O
The	O
first	O
word	O
cloud	O
contains	O
top	O
three	O
words	O
which	O
gain	O
the	O
highest	O
positive	O
relevance	O
scores	O
from	O
each	O
training	O
example	O
,	O
while	O
the	O
second	O
word	O
cloud	O
does	O
the	O
same	O
but	O
for	O
the	O
top	O
three	O
words	O
which	O
gain	O
the	O
lowest	O
negative	O
relevance	O
scores	O
(	O
see	O
Figure	O
11	O
)	O
.	O

Furthermore	O
,	O
we	O
also	O
conducted	O
Experiment	O
1	O
for	O
BiLSTMs	O
.	O
Each	O
direction	O
of	O
the	O
recurrent	O
layer	O
had	O
15	O
hidden	O
units	O
and	O
the	O
feature	O
vector	O
was	O
obtained	O
by	O
taking	O
element	O
-	O
wise	O
max	O
of	O
all	O
the	O
hidden	O
states	O
(	O
i.e.	O
,	O
d	O
=	O
15	O
×	O
2	O
=	O
30	O
)	O
.	O
We	O
adapted	O
the	O
code	O
of	O
(	O
Arras	O
et	O
al	O
.	O
,	O
2017	O
)	O
to	O
run	O
LRP	O
on	O
BiLSTMs	O
.	O
Regarding	O
human	O
feedback	O
collection	O
,	O
we	O
collected	O
feedback	O
from	O
Amazon	O
Mechanical	O
Turk	O
workers	O
by	O
splitting	O
the	O
pair	O
of	O
word	O
clouds	O
into	O
two	O
and	O
asking	O
the	O
question	O
about	O
the	O
relevant	O
class	O
independently	O
of	O
each	O
other	O
.	O
The	O
answer	O
of	O
the	O
positive	O
relevance	O
word	O
cloud	O
should	O
be	O
consistent	O
with	O
the	O
weight	O
matrix	O
W	O
,	O
while	O
the	O
answer	O
of	O
the	O
negative	O
relevance	O
word	O
cloud	O
should	O
be	O
the	O
opposite	O
of	O
the	O
weight	O
matrix	O
W.	O
The	O
score	O
of	O
a	O
BiLSTM	O
feature	O
is	O
the	O
sum	O
of	O
its	O
scores	O
from	O
the	O
positive	O
word	O
cloud	O
and	O
the	O
negative	O
word	O
cloud	O
.	O

The	O
results	O
of	O
the	O
extra	O
BiLSTM	O
experiments	O
are	O
shown	O
in	O
Table	O
4	O
and	O
5	O
.	O
Table	O
4	O
shows	O
unexpected	O
results	O
after	O
disabling	O
features	O
.	O
For	O
instance	O
,	O
disabling	O
rank	O
B	O
features	O
caused	O
a	O
larger	O
performance	O
drop	O
than	O
removing	O
rank	O
A	O
features	O
.	O
This	O
suggests	O
that	O
how	O
we	O
created	O
word	O
clouds	O
for	O
each	O
BiLSTM	O
feature	O
(	O
i.e.	O
,	O
displaying	O
top	O
three	O
words	O
with	O
the	O
highest	O
positive	O
and	O
lowest	O
negative	O
rel-	O
evance	O
)	O
might	O
not	O
be	O
an	O
accurate	O
way	O
to	O
explain	O
the	O
feature	O
.	O
Nevertheless	O
,	O
another	O
observation	O
from	O
Table	O
4	O
is	O
that	O
even	O
when	O
we	O
disabled	O
two	O
-	O
third	O
of	O
the	O
BiLSTM	O
features	O
,	O
the	O
maximum	O
macro	O
F1	O
drop	O
was	O
less	O
than	O
5	O
%	O
.	O
This	O
suggests	O
that	O
there	O
is	O
a	O
lot	O
of	O
redundant	O
information	O
in	O
the	O
features	O
of	O
the	O
BiLSTMs	O
.	O

D	O
Metrics	O
for	O
Biases	O

In	O
this	O
paper	O
,	O
we	O
used	O
two	O
metrics	O
to	O
quantify	O
biases	O
in	O
the	O
models	O
-False	O
positive	O
equality	O
difference	O
(	O
FPED	O
)	O
and	O
False	O
negative	O
equality	O
difference	O
(	O
FNED	O
)	O
-with	O
the	O
following	O
definitions	O
(	O
Dixon	O
et	O
al	O
.	O
,	O
2018	O
)	O
.	O

F	O
P	O
ED	O
=	O
t∈T	O
|F	O
P	O
R	O
−	O
F	O
P	O
R	O
t	O
|	O
F	O
N	O
ED	O
=	O
t∈T	O
|F	O
N	O
R	O
−	O
F	O
N	O
R	O
t	O
|	O

where	O
T	O
is	O
a	O
set	O
of	O
all	O
sub	O
-	O
populations	O
we	O
consider	O
(	O
i.e.	O
,	O
T	O
=	O
{	O
male	O
,	O
female	O
}	O
)	O
.	O
FPR	O
and	O
FNR	O
stand	O
for	O
false	O
positive	O
rate	O
and	O
false	O
negative	O
rate	O
,	O
respectively	O
.	O
The	O
subscript	O
t	O
means	O
that	O
we	O
calculate	O
the	O
metrics	O
using	O
data	O
examples	O
mentioning	O
the	O
sub	O
-	O
population	O
t	O
only	O
.	O
We	O
used	O
the	O
following	O
keywords	O
to	O
identify	O
examples	O
which	O
are	O
related	O
to	O
or	O
mentioning	O
the	O
sub	O
-	O
populations	O
.	O
Male	O
gender	O
terms	O
:	O

"	O
male	O
"	O
,	O
"	O
males	O
"	O
,	O
"	O
boy	O
"	O
,	O
"	O
boys	O
"	O
,	O
"	O
man	O
"	O
,	O
"	O
men	O
"	O
,	O
"	O
gentleman	O
"	O
,	O
"	O
gentlemen	O
"	O
,	O
"	O
he	O
"	O
,	O
"	O
him	O
"	O
,	O
"	O
his	O
"	O
,	O
"	O
himself	O
"	O
,	O
"	O
brother	O
"	O
,	O
"	O
son	O
"	O
,	O
"	O
husband	O
"	O
,	O
"	O
boyfriend	O
"	O
,	O
"	O
father	O
"	O
,	O
"	O
uncle	O
"	O
,	O
"	O
dad	O
"	O
Female	O
gender	O
terms	O
:	O

"	O
female	O
"	O
,	O
"	O
females	O
"	O
,	O
"	O
girl	O
"	O
,	O
"	O
girls	O
"	O
,	O
"	O
woman	O
"	O
,	O
"	O
women	O
"	O
,	O
"	O
lady	O
"	O
,	O
"	O
ladies	O
"	O
,	O
"	O
she	O
"	O
,	O
"	O
her	O
"	O
,	O
"	O
herself	O
"	O
,	O
"	O
sister	O
"	O
,	O
"	O
daughter	O
"	O
,	O
"	O
wife	O
"	O
,	O
"	O
girlfriend	O
"	O
,	O
"	O
mother	O
"	O
,	O
"	O
aunt	O
"	O
,	O
"	O
mom	O
"	O
.	O
All	O
the	O
bios	O
are	O
from	O
Common	O
Crawl	O
August	O
2018	O
Index	O
.	O

•	O
Waseem	O
:	O
The	O
authors	O
of	O
(	O
Waseem	O
and	O
Hovy	O
,	O
2016	O
)	O
kindly	O
provided	O
the	O
dataset	O
to	O
us	O
by	O
email	O
.	O
We	O
considered	O
"	O
racism	O
"	O
and	O
"	O
sexism	O
"	O
examples	O
as	O
"	O
Abusive	O
"	O
and	O
"	O
neither	O
"	O
examples	O
as	O
"	O
Non	O
-	O
abusive	O
"	O
.	O

•	O
Wikitoxic	O
:	O
The	O
dataset	O
can	O
be	O
downloaded	O
here	O
10	O
.	O
We	O
used	O
only	O
examples	O
which	O
were	O
given	O
the	O
same	O
label	O
by	O
all	O
the	O
annotators	O
.	O

•	O
20Newsgroups	O
:	O
We	O
downloaded	O
the	O
standard	O
splits	O
of	O
the	O
dataset	O
using	O
scikit	O
-	O
learn	O
11	O
.	O
The	O
header	O
and	O
the	O
footer	O
of	O
each	O
text	O
were	O
removed	O
.	O

F	O
Full	O
Experimental	O
Results	O

Model	O
:	O
CNNs	O

Test	O
dataset	O
:	O
Yelp	O
Negative	O
F1	O

Positive	O
F1	O
Accuracy	O
Macro	O
F1	O
Original	O
0.767	O
±	O
0.02	O
0.800	O
±	O
0.00	O
0.785	O
±	O
0.01	O
0.789	O
±	O
0.01	O
Disabling	O
(	O
MTurk	O
)	O
0.786	O
±	O
0.00	O
0.804	O
±	O
0.00	O
0.795	O
±	O
0.00	O
0.796	O
±	O
0.00	O

Acknowledgments	O

We	O
would	O
like	O
to	O
thank	O
Nontawat	O
Charoenphakdee	O
and	O
anonymous	O
reviewers	O
for	O
helpful	O
comments	O
.	O
Also	O
,	O
the	O
first	O
author	O
wishes	O
to	O
thank	O
the	O
support	O
from	O
Anandamahidol	O
Foundation	O
,	O
Thailand	O
.	O