Collecting Entailment Data for Pretraining:New Protocols and Negative Results
Table 6: Model performance on the SuperGLUE validation and diagnostic sets. The Avg. column shows the overall SuperGLUE score—an average across the eight tasks, weighting each task equally—as a mean and standard deviation across three restarts.
['[BOLD] Intermediate-  [BOLD] Training Data', '[BOLD] Avg.  [ITALIC] μ ( [ITALIC] σ)', '[BOLD] BoolQ  [BOLD] Acc.', '[BOLD] CB  [BOLD] F1/Acc.', '[BOLD] CB  [BOLD] F1/Acc.', '[BOLD] COPA  [BOLD] Acc.', '[BOLD] MultiRC  [BOLD] F1 [ITALIC] a/EM', '[BOLD] MultiRC  [BOLD] F1 [ITALIC] a/EM', '[BOLD] ReCoRD  [BOLD] F1/EM', '[BOLD] ReCoRD  [BOLD] F1/EM', '[BOLD] RTE  [BOLD] Acc.', '[BOLD] WiC  [BOLD] Acc.', '[BOLD] WSC  [BOLD] Acc.']
Our first observation is that our overall data collection pipeline worked well for our purposes: Our Base data yields models that transfer substantially better than the plain RoBERTa or XLNet baseline, and at least slightly better than 8.5k-example samples of MNLI, MNLI Government or ANLI.