Don’t Take the Easy Way Out:Ensemble Based Methods for Avoiding Known Dataset Biases
Table 4: Accuracy on the adversarial MNLI dataset, HANS, and the MNLI matched dev set.
['Debiasing Method', 'Co-Attention HANS', 'Co-Attention MNLI', 'BERT HANS', 'BERT MNLI']
Results: For the recurrent method, both the bias product and learned-mixin +H methods result in about a three point gain. However, for the BERT model, the simpler reweight method is more effective. We noticed high variance in performance between runs in this setting, and speculate the ensemble methods might be compounding this instability by introducing additional complexity.