OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings
Table 2: Extrinsic Bias Measurement: Gender bias expressed downstream by GloVE and RoBERTa embeddings as observed using NLI as a probe for bias measurement.
['Embedding', 'Net Neutral', 'Fraction Neutral', 'Dev', 'Test']
All debiasing methods, including OSCaR, increase the neutrality score (recall sentences should be objectively neutral) without significantly decreasing the performance (Dev/Test scores); HD on GloVe is the only one where these scores drop more than 0.015. While INLP appears to be the most successful at debiasing GloVe, OSCaR and LP are clear the most successful on RoBERTa (with LP slightly better); this is likely since they are differentiable, and can be integrated into fine-tuning. Using RoBERTa, we see a similar pattern where OSCaR outperforms the projection based methods, LP and HD in the retention of valid gendered information. It also does better than INLP on the entailment tasks. However, INLP does better on the contradition tasks than OSCaR.