Does Gender Matter? Towards Fairness in Dialogue Systems
Table 8: Fairness test of the debiased Seq2Seq generative model. Green value indicates that the absolute value of difference drops compared with the original model, while red value indicates it rises.
['[EMPTY]', '[BOLD] Gender  [BOLD] CDA', '[BOLD] Gender  [BOLD] CDA', '[BOLD] Gender  [BOLD] CDA', '[BOLD] Gender  [BOLD] CDA', '[BOLD] Gender  [BOLD] WER', '[BOLD] Gender  [BOLD] WER', '[BOLD] Gender  [BOLD] WER', '[BOLD] Gender  [BOLD] WER']
We can observe that first, for most of the cases, both of the two debiasing models reduce gender biases and race biases in terms of various measurements significantly. The differences between the two groups are controlled within a reasonable range and are not statistically significant anymore. Second, WER performs better than CDA in mitigating biases. However, a drawback of WER is, after sufficient training with the regularization term, the dialogue model tends to generate similar responses to two genders/races, which may debase the diversity of the generated responses. It reminds us that there may exist a trade-off between the performance and the fairness of a model. It’s important for us to find a balance according to specific situations.