Don’t Take the Easy Way Out:Ensemble Based Methods for Avoiding Known Dataset Biases
Table 8: Scores on individual heuristics in HANS, with scores on the MNLI matched dev set for reference. Results are an average of 8 runs.
['Model', 'Debiasing Method', 'MNLI', 'Correct:  [ITALIC] Entailment Lexical', 'Correct:  [ITALIC] Entailment Subseq.', 'Correct:  [ITALIC] Entailment Const', 'Correct:  [ITALIC] Non-entailment Lexical', 'Correct:  [ITALIC] Non-entailment Subseq.', 'Correct:  [ITALIC] Non-entailment Const']
Our methods reduce the extent to which models naively guess entailment in all cases. Interestingly, the BERT model shows significantly degraded performance on the entailment examples when using the reweight and bias product method, but largely maintains its performance on those examples when using the learned-mixin method.