HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference
Table 7: The comparison of InferSent (baseline), InferSent+DS (downsampling) and dInferSent (adversarial debiasing) on SNLI test set and MultiNLI mismatched dev set respectively. We choose the down-sampling (DS) method with λ=0.8 because it performs best on the hard subsets. The ‘Guidance’ and ‘Reweight’ methods are elaborated in Sec 3.2.2 and Sec 3.2.3 respectively.
['[BOLD] Model', '[BOLD] Full', '[BOLD] Easy', '[BOLD] Hard', 'Δ [ITALIC] HardEasy(↓)']
Parameter settings: Specifically, we treat the instances which contain the artificial patterns in H(3,3,λ) For a fair comparison, we do not tune any hyper-parameter in the InferSent encoder, the learning rate and the optimizer setting. , we observe that although the performance gap between the easy and hard subsets is reduced to some extent by the vanilla dInferSent models in both SNLI and MultiNLI. The model still does not reach our expectation to lower the gap between hard and easy sets. The explicit guidance (‘+Guidance’) from the artificial patterns alleviates this issue in both datasets as in this way the discriminator pays more attention on the potentially biased instances thus has smaller influence on the hard instances in the training procedure. These models achieve higher accuracies on the hard subset than the baseline models in both datasets.