Collecting Entailment Data for Pretraining:New Protocols and Negative Results
Table 4: NLI modeling experiments with RoBERTa, reporting results on the validation sets for MNLI and for the task used for training each model (Self), and the GLUE diagnostic set. We compare the two-class Contrast with a two-class version of MNLI.
['[BOLD] Training Data', '[BOLD] Self', '[BOLD] MNLI', '[BOLD] GLUE Diag.']
As NLI classifiers trained on Contrast cannot produce the neutral labels used in MNLI, we evaluate them separately, and compare them with two-class variants of the MNLI models. Our first three interventions, especially EditPremise, show much lower hypothesis-only performance than Base. This indicates that these these results cannot be explained away as a consequence of the lower quality of the evaluation sets for these three new datasets. This adds further evidence, alongside our PMI results, that these interventions reduce the presence of such artifacts. While we do not have a direct baseline for the two-class Contrast in this experiment, comparisons with MNLI 8.5k are consistent with the encouraging results seen above.