Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function
Table 3: Evaluation results for models trained on Daily Mail and their generated texts
['Model', '[ITALIC] BN', '[ITALIC] BNc', '[ITALIC] GR', '[ITALIC] Ppl.', '[ITALIC] CB| [ITALIC] o', '[ITALIC] CB| [ITALIC] g', '[ITALIC] EBd']
Initially, we measure the co-occurrence bias in the training data. After training the baseline model, we implement our loss function and tune for the λ hyperparameter. Additionally, we implement a combination of our loss function and CDA and tune for λ. Finally, bias evaluation is performed for all the trained models. Causal occupation bias is measured directly from the models using template datasets discussed above and co-occurrence bias is measured from the model-generated texts, which consist of 10,000 documents of 500 words each. It is interesting to observe that the baseline model amplifies the bias in the training data set as measured by BNand BNc. From measurements using the described bias metrics, our method effectively mitigates bias in language modelling without a significant increase in perplexity. At λ value of 1, it reduces BN by 58.95%, BNc by 45.74%, CB|o by 100%, CB|g by 98.52% and EBd by 98.98%. Compared to the results of CDA and REG, it achieves the best results in both occupation biases, CB|g and CB|o, and EBd. We notice that all methods result in GR around 1, indicating that there are near equal amounts of female and male words in the generated texts. In our experiments we note that with increasing λ, the bias steadily decreases and perplexity tends to slightly increase. This indicates that there is a trade-off between bias and perplexity.