Reducing Sentiment Bias in Language Modelsvia Counterfactual Evaluation
Table 1: Perplexity and semantic similarity scores of WMT19 and WikiText-103 models for the Country attribute. A lower perplexity is better; higher semantic similarity scores (S.S. and S.S.c) are better.
['[EMPTY]', '[EMPTY]', 'WMT-19  [ITALIC] Country', 'WMT-19  [ITALIC] Country', 'WMT-19  [ITALIC] Country', 'WMT-19  [ITALIC] Country', 'WikiText-103  [ITALIC] Country', 'WikiText-103  [ITALIC] Country', 'WikiText-103  [ITALIC] Country', 'WikiText-103  [ITALIC] Country']
We observe the model could generate irrelevant sentences if trained using a very large debiasing regularization parameter λ. In this case, the model would be “fair” in the sense that it completely ignores the sensitive attribute tokens. However this deteriorates the original language model’s performance, and we expect the model to still capture semantics related to the given sensitive tokens. Thus, it is important to evaluate the trade-off between generation quality and fairness by observing perplexity (PPL and PPLs) and semantic similarity scores (S.S. and S.S.c). We present these metrics for the same set of models as in Figures We can also observe that a larger regularization reduces semantic similarity scores (the trends are similar for both S.S. and S.S.c).