Adversarial NLI: A New Benchmark for Natural Language Understanding
Table 6: Performance of RoBERTa with different data combinations. ALL=S,M,F,ANLI. Hypothesis-only models are marked H where they are trained and tested with only hypothesis texts.
['[BOLD] Training Data', 'A1', 'A2', 'A3', 'S', 'M-m/mm']
Even though ANLI training data is different from SNLI and MNLI, adding it to the training set does not harm performance on those tasks. For SNLI and MNLI, concerns have been raised about the propensity of models to pick up on spurious artifacts that are present just in the hypotheses Gururangan et al. ; Poliak et al. Here, we compare full models to models trained only on the hypothesis (marked H). Hypothesis-only performance decreases over rounds for ANLI. We observe that in rounds 2 and 3, RoBERTa is not much better than hypothesis-only. This could mean two things: either the test data is very difficult, or the training data is not good. To rule out the latter, we trained only on ANLI (∼163k training examples): Hence, this shows that the test sets are so difficult that state-of-the-art models cannot outperform a hypothesis-only prior.