Reducing Sentiment Bias in Language Modelsvia Counterfactual Evaluation
Table 5: Perplexity and semantic similarity scores of WMT19 and WikiText-103 models for the Occupation attribute. A lower perplexity is better; higher semantic similarity scores (S.S. and S.S.c) are better.
['[EMPTY]', '[EMPTY]', 'WMT-19  [ITALIC] Occupation', 'WMT-19  [ITALIC] Occupation', 'WMT-19  [ITALIC] Occupation', 'WMT-19  [ITALIC] Occupation', 'WikiText-103  [ITALIC] Occupation', 'WikiText-103  [ITALIC] Occupation', 'WikiText-103  [ITALIC] Occupation', 'WikiText-103  [ITALIC] Occupation']
Perplexity on the test set (PPL) and the subset of the test set that contains sensitive tokens (PPLs) remain almost unchanged, however the semantic similarities between the sensitive token and the generated texts can be decreased when the regularization parameter is too large. Note that since we do not further train our baseline model with the additional epochs of the debiasing step, both perplexity (PPL and PPLs) can sometimes improve, while improving individual fairness and group fairness measures.