MDR Cluster-Debias: A Nonlinear Word Embedding Debiasing Pipeline
Table 1: Results for our upstream bias evaluations. The first three rows are extracted directly from prior work [Gonen2019LipstickOA]. The last four rows are the debiased word embeddings using our proposed pipeline. Bolded results are the best-performing in each column according to the given metric.
['[BOLD] Embedding', '[BOLD] Kmeans Acc.', '[BOLD] Corr. Prof.', '[BOLD] SVM Acc.', '[BOLD] Work/Family P-val', '[BOLD] Math/Art 1 P-val', '[BOLD] Math/Art 2 P-val']
Results can be summarized as follows: Cluster-GloVe outperforms GN-Glove and Hard-GloVe on all cluster-bias based tests because of following reasons. First, Post-debias Cluster Accuracy of Cluster-GloVe is 0.53, which means that debiased gendered words are not separable by K-means. Cluster-GloVe is also the most difficult to classify post-hoc using an SVM (it has lowest SVM Classifier Accuracy). And it shows no gender bias on two of the three WEAT tests ( p-values of last two columns show no significant results). With respect to deficiencies in the Cluster-GloVe, embeddings still are highly separable by SVM, and as evidenced from the correlational professions experiment and the Work/Family WEAT, retain gender stereotypes for occupations. Our Post-Processing method is able to help not only the Cluster-Debias method but also HardDebias. MDR-Cluster and MDR-Hard outperform Cluster-GloVe and Hard-GloVe respectively. MDR-Cluster achieves the best overall performance and is the only method that prevents SVM from classifying gender stereotyped words, which validates the efficiency of our debias pipeline. But MDR-Cluster still struggles with work/family associations WEAT test.