On Measuring and Mitigating Biased Inferences of Word Embeddings
Table 16: SNLI dev/test accuracies before debiasing ELMo (orig) and after debiasing gender on all layers and layer 1, debiasing nationality and religions on layer 1.
['SNLI Accuracies (ELMo)', 'SNLI Accuracies (ELMo) orig', 'SNLI Accuracies (ELMo) -gen(all)', 'SNLI Accuracies (ELMo) -gen(1)', 'SNLI Accuracies (ELMo) -nat(1)', 'SNLI Accuracies (ELMo) -rel(1)']
We see that the effect of debiasing is fairly minor on the original prediction goal, and these scores remain slightly larger than the models based on GloVe, both before and after debiasing. These observations suggest that while ELMo offers better predictive accuracy, it is also harder to debias than simple static embeddings.