How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Table 1: The performance of various static embeddings on word embedding benchmark tasks. The best result for each task is in bold. For the contextualizing models (ELMo, BERT, GPT-2), we use the first principal component of a word’s contextualized representations in a given layer as its static embedding. The static embeddings created using ELMo and BERT’s contextualized representations often outperform GloVe and FastText vectors.
['Static Embedding', 'SimLex999', 'MEN', 'WS353', 'RW', 'Google', 'MSR', 'SemEval2012(2)', 'BLESS', 'AP']
As noted earlier, we can create static embeddings for each word by taking the first principal component (PC) of its contextualized representations in a given layer. These tasks cover semantic similarity, analogy solving, and concept categorization: SimLex999 hill2015simlex, MEN bruni2014multimodal, WS353 finkelstein2002placing, RW luong2013better, SemEval-2012 jurgens2012semeval, Google analogy solving mikolov2013distributed MSR analogy solving mikolov2013linguistic , BLESS baroni2011we and AP almuhareb2004attribute.