Improving Natural Language Inference with a Pretrained Parser
Table 5: Model accuracies on HANS.
['Model', 'Train Data', 'Correct:  [ITALIC] Entailment Lexical', 'Correct:  [ITALIC] Entailment Subseq.', 'Correct:  [ITALIC] Entailment Const.', 'Correct:  [ITALIC] Non-entailment Lexical', 'Correct:  [ITALIC] Non-entailment Subseq.', 'Correct:  [ITALIC] Non-entailment Const.', 'Avg.']
We evaluate a subset of our models on HANS. In particular, we choose the model/dataset combinations whose test accuracies improve the most with SWRs: DA+SA on SciTail (+3.1%), ESIM+SA on SciTail (+5.1%), BERT+LF on SciTail (+2.0%), and MT-DNN+LF on MNLI-mm (+0.5%). We find SWRs improve overall accuracy on HANS (1–3%) over their non-SWR counterparts for all four models listed above. We also observe that SWRs significantly reduce the models’ reliance on fallible syntactic heuristics—that is, the accuracies between the entailment and non-entailment labels are far more balanced.