Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual
Table 5: F1 scores of each class on Stress. Intensity of the Blue and red highlights corresponds to absolute increase and decrease of scores with respect to MLE. DRiFt improves results on E (that exhibits label shift) with some degradation on other classes for DA and ESIM.
['method', 'Negation  [ITALIC] E', 'Negation  [ITALIC] C', 'Negation  [ITALIC] N', 'Overlap  [ITALIC] E', 'Overlap  [ITALIC] C', 'Overlap  [ITALIC] N']
Compared to results on HANS, Stress sees lower overall improvement from debiasing. One reason is that Stress decreases word overlap rate and injects negation words by appending distractor phrases, i.e. “true is true” and “false is not true”. While this introduces label shift on biased features, it also introduces covariate shift on the input. For example, although Hand contains features designed to use word overlap rate (Jaccard similarity) and negation words, its does not have big performance drop on the challenge data compared to its in-distribution performance, showing that that distractor phrases may affect the model in other ways.