End-to-End Bias Mitigation by Modelling Biases in Corpora
Table 9: Accuracy results of models with InferSent transferring to new target datasets. All models are trained on SNLI and tested on the target datasets. M1 and M2 are our re-implementation of belinkov-etal-2019-dont. Δ are relative differences in percentage with respect to CE loss.
['[BOLD] Data SICK', '[BOLD] CE 54.09', '[BOLD] DFL 55.00', '[BOLD] Δ% 1.68', '[BOLD] PoE 55.79', '[BOLD] Δ% 3.14', '[BOLD] M1 49.77', '[BOLD] Δ% -7.99', '[BOLD] M2 49.77', '[BOLD] Δ% -7.99']
As used in prior work to adjust the learning-rate of the bias-only and baseline models (belinkov-etal-2019-dont), we introduce a hyperparameter β for the bias-only model to modulate the loss of the bias-only model in ensembling. We sweep hyper-parameters γ,α over {0.02,0.05,0.1,0.6,2.0,4.0,5.0} and β over {0.05,0.2,0.4,0.8,1.0}. The DFL model outperforms the baseline in 10 out of 12 datasets, while the PoE model outperforms the baseline in 9 datasets and does equally well on the DPR dataset. As shown in prior work (belinkov-etal-2019-dont), the MNLI dataset has very similar biases to SNLI, which the models are trained on, so we do not expect any improvement in the relative performance of our models and the baseline for MNLI dataset. Interestingly, our methods obtain improvement on MNLI-M, in which the test data differs from training distribution. Our proposed debiasing methods, PoE and DFL, are highly effective, boosting the relative generalization performance of the baseline by 3.39% and 2.57% respectively, significantly surpassing the prior work of belinkov-etal-2019-dont. Compared to M1 and M2, our methods outperform them on 9 datasets, while they do better on two datasets of SPR and FN+, and slightly better on the DPR dataset. However, note that DPR is a very small dataset and all models perform close to random-chance on this dataset.