Annotation Artifacts in Natural Language Inference Data
Table 5: Performance of high-performing NLI models on the full, Hard, and Easy NLI test sets.
['[BOLD] Model', '[BOLD] SNLI  [ITALIC] Full', '[BOLD] SNLI  [ITALIC] Hard', '[BOLD] SNLI  [ITALIC] Easy', '[BOLD] MultiNLI Matched  [ITALIC] Full', '[BOLD] MultiNLI Matched  [ITALIC] Hard', '[BOLD] MultiNLI Matched  [ITALIC] Easy', '[BOLD] MultiNLI Mismatched  [ITALIC] Full', '[BOLD] MultiNLI Mismatched  [ITALIC] Hard', '[BOLD] MultiNLI Mismatched  [ITALIC] Easy']
While the models correctly classify some Hard examples, the bulk of their success is attributed to the Easy examples. This result implies that the ability of NLI models to recognize textual entailment is lower than previously perceived, and that such models rely heavily on annotation artifacts in the hypothesis to make their predictions.