bert.conll

MNLI B-DatasetName
Multi B-DatasetName
- I-DatasetName
Genre I-DatasetName
Natural I-DatasetName
Language I-DatasetName
Inference I-DatasetName
is O
a O
large O
- O
scale, O
crowdsourced O
entailment B-TaskName
classification I-TaskName
task I-TaskName
( O
Williams O
et O
al. O
, O
2018 O
) O
. O
Given O
a O
pair O
of O
sentences O
, O
the O
goal O
is O
to O
predict O
whether O
the O
second O
sentence O
is O
an O
entailment O
, O
contradiction O
, O
or O
neutral O
with O
respect O
to O
the O
first O
one O
. O

F1 B-MetricName
scores O
are O
reported O
for O
QQP B-DatasetName
and O
MRPC B-DatasetName
, O
Spearman B-MetricName
correlations I-MetricName
are O
reported O
for O
STS B-DatasetName
- I-DatasetName
B I-DatasetName
, O
and O
accuracy B-MetricName
scores O
are O
reported O
for O
the O
other O
tasks O
. O
We O
exclude O
entries O
that O
use O
BERT B-MethodName
as O
one O
of O
their O
components O
. O

In O
this O
work O
, O
we O
denote O
the O
number B-HyperparameterName
of I-HyperparameterName
layers I-HyperparameterName
( O
i.e. O
, O
Transformer O
blocks O
) O
as O
L B-HyperparameterName
, O
the O
hidden B-HyperparameterName
size I-HyperparameterName
as O
H B-HyperparameterName
, O
and O
the O
number B-HyperparameterName
of I-HyperparameterName
self I-HyperparameterName
- I-HyperparameterName
attention I-HyperparameterName
heads I-HyperparameterName
as O
A B-HyperparameterName
. O
We O
primarily O
report O
results O
on O
two O
model O
sizes O
: O
BERTBASE B-MethodName
( O
L B-HyperparameterName
= O
12 B-HyperparameterValue
, O
H B-HyperparameterName
= O
768 B-HyperparameterValue
, O
A B-HyperparameterName
= O
12 B-HyperparameterValue
, O
Total O
Parameters O
= O
110M O
) O
and O
BERTLARGE B-MethodName
( O
L B-HyperparameterName
= O
24 B-HyperparameterValue
, O
H B-HyperparameterName
= O
1024 B-HyperparameterValue
, O
A B-HyperparameterName
= O
16 B-HyperparameterValue
, O
Total O
Parameters O
= O
340M O
) O
. O

In O
this O
paper O
, O
we O
improve O
the O
fine O
- O
tuning O
based O
approaches O
by O
proposing O
BERT B-MethodName
: O
Bidirectional B-MethodName
Encoder I-MethodName
Representations I-MethodName
from I-MethodName
Transformers I-MethodName
. O
BERT B-MethodName
alleviates O
the O
previously O
mentioned O
unidirectionality O
constraint O
by O
using O
a O
“ O
masked O
language O
model O
” O
( O
MLM O
) O
pre O
- O
training O
objective O
, O
inspired O
by O
the O
Cloze O
task O
( O
Taylor O
, O
1953 O
) O
. O

On O
the O
official O
GLUE B-DatasetName
leaderboard O
, O
BERTLARGE B-MethodName
obtains O
a O
score O
of O
80.5 B-MetricValue
, O
compared O
to O
OpenAI B-MethodName
GPT, I-MethodName
which O
obtains O
72.8 B-MetricValue
as O
of O
the O
date O
of O
writing O
. O