Unsupervised Discovery of Implicit Gender Bias
Table 3: Evaluation over held-out test set for each data set, where w_gender = F is considered the positive class. Latent confound demotion generally improves performance.
['[EMPTY]', '[BOLD] Prec.', '[BOLD] Rec.', '[BOLD] F1', '[BOLD] Acc.']
We created data splits so that there is no overlap in ow values between the train and test sets. While there may still be overlap in some latent w_traits, we expect there to be less overlap in w_traits between the train and test set than within the train set. Thus, improved performance over the held-out test set would suggest that demotion effectively reduces the influence of the latent confounding variables. We do not necessarily expect propensity matching to improve performance, as this method reduces the influence of confounding variables that have high overlap between the train and test sets. As expected, models with demotion perform best on all metrics, with the exception of recall in the Politicians data. The discrepencies between F1 and Accuracy are explained by the imbalance in the data set, particularly in the Politicians data set, which is imbalanced in favor of M while we report metrics assuming F is the positive class.