OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings
Table S2: Intrinsic Standard Information Tests: These standard tests evaluate the amount of overall coherent associations in word embeddings. WSim is a word similarity test and Google Analogy is a set of analogy tests.
['Embedding', 'GloVe', 'GloVe LP', 'GloVe OSCaR', 'GloVe INLP', 'GloVe HD']
Non-contextual word embeddings like GloVe or word2vec are characterized by their ability to capture semantic information reflected by valid similarities between word pairs and the ability to complete analogies. These properties reflect the quality of the word embedding obtained and should not be diminished post debiasing. We use a standard word similarity test [wsim] and an analogy test [wordtovec] which measure these properties in word embeddings across our baseline GloVe model and all the debiased GloVe models. All of them perform similarly to the baseline GloVe model, thus indicating that the structure of the embedding has been preserved. While this helps in preliminary examination of the retention of information in the embeddings, these tests do not contain a large number of sensitive gender comparisons or word pairs. It is thus, not sufficient to claim using this that bias has been removed or that valid gender associations have been retained, leading to the requirement of methods described in the paper and methods in other work [Bias1, bias2, debias, Caliskan183] for the same.