Gender-preserving Debiasing for Pre-trained Word Embeddings
Table 1: Prediction accuracies for gender relational analogies. ∗ and † indicate statistically significant differences against respectively GloVe and Hard-GloVe.
['Embeddings', 'SemBias Definition ↑', 'SemBias Stereotype ↓', 'SemBias None ↓', 'SemBias-subset Definition ↑', 'SemBias-subset Stereotype ↓', 'SemBias-subset None ↓']
We use the SemBias dataset created by \newciteZhao:2018ab to evaluate the level of gender bias in word embeddings. Each instance in SemBias consists of four word pairs: a gender-definition word pair (Definition; e.g. “waiter - waitress”), a gender-stereotype word pair (Stereotype; e.g., “doctor - nurse”) and two other word-pairs that have similar meanings but not a gender relation (None; e.g., “dog - cat”, “cup - lid”). SemBias contains 20 gender-stereotype word pairs and 22 gender-definitional word pairs and use their Cartesian product to generate 440 instances. Among the 22 gender-definitional word pairs, 2 word-pairs are not used as the seeds for training. Following, \newciteZhao:2018ab, to test the generalisability of a debiasing method, we use the subset (SemBias-subset) of 40 instances associated with these 2 pairs. We measure relational similarity between (he,she) word-pair and a word-pair (a,b) in SemBias using the cosine similarity between the \vvhe−\vvshe gender directional vector and →a−→b using the word embeddings under evaluation. For the four word-pairs in each instance in SemBias, we select the word-pair with the highest cosine similarity with \vvhe−\vvshe as the predicted answer. If the word embeddings are correctly debiased, we would expect a high accuracy for Definitions and low accuracies for Stereotypes and Nones. Although GN-GloVe obtains high performance on SemBias, it does not generalise well to SemBias-subset. However, by applying the proposed method, we can further remove any residual gender biases from GN-GloVe, which shows that the proposed method can be applied in conjunction with GN-GloVe. We see that GloVe contains a high percentage of stereotypical gender biases, which justifies the need for debiasing methods. By applying the proposed method on GloVe (corresponds to GP (GloVe)) we can decrease the gender biases in GloVe, while preserving useful gender-related information for detecting definitional word-pairs. Comparing corresponding AE and GP versions of GloVe and GN-GloVe, we see that autoencoding alone is insufficient to consistently preserve gender-related information.