On Measuring and Mitigating Biased Inferences of Word Embeddings
Table 12: Fraction of the top principal value with the xth principal value with the ELMo layer 1 embedding for Gendered, Demonym, and Adherent datasets. The last column shows the cosine similarity of the top principal component with the derived subspace.
['Embedding', '2nd', '3rd', '4th', 'cosine']
Next, we show how to significantly attenuate gender bias in ELMo embeddings: we invoke the projection mechanism, but only on layer 1. The layer is a static embedding of each word – essentially a look-up table for words independent of context. Thus, as with GloVe we can find a strong subspace for gender using only the he-she vector.