Stress Test Evaluation for Natural Language Inference
Table 4: % of false neutral in total errors on MultiNLI development set, word overlap test and length mismatch test.
['[BOLD] Sys  [BOLD] tem', '[BOLD] MultiNLI  [BOLD] Dev', '[BOLD] MultiNLI  [BOLD] Dev', '[BOLD] Word  [BOLD] Overlap', '[BOLD] Word  [BOLD] Overlap', '[BOLD] Length  [BOLD] Mismatch', '[BOLD] Length  [BOLD] Mismatch']
The significant decrease in accuracy indicates that lexical similarity is a strong signal for entailment prediction, failing which models default to predicting neutral. To provide further justification, we compare the proportion of false neutral errors for all models on word overlap and length mismatch stress sets vs. the original MultiNLI development set.