Adversarial Analysis of Natural Language Inference Systems
TABLE II: Single Word Replacement Attacks from [7]. *ESIM and KIM results from original paper.
['Model', 'Accuracy (%)']
This dataset is created by modifying SNLI examples with single word replacements of different lexical relations, based on WordNet. It tests lexical inferences and relatively simple world knowledge. They test a model called KIM (Knowledge-based Inference Model) However, the BERT-based models still significantly outperform KIM. This could be due to model architecture, but is most likely a result of their extensive pretraining on a huge diverse corpus. There is not a big difference between model sizes, or between MT-DNN and BERT.