How Can BERT Help Lexical Semantics Tasks?
Table 1: Main results on word similarity and analogy tasks. The ELMo, GPT2, BERT and XLNet models use 512, 768, 768 and 768 dimensional embeddings, respectively, while others use 300 dimensional vectors.
['[BOLD] Types', '[BOLD] Models', '[BOLD] Word Similarity  [BOLD] WS353', '[BOLD] Word Similarity  [BOLD] WS353S', '[BOLD] Word Similarity  [BOLD] WS353R', '[BOLD] Word Similarity  [BOLD] SimLex-999', '[BOLD] Word Similarity  [BOLD] RW', '[BOLD] Word Similarity  [BOLD] MEN', '[BOLD] Analogy  [BOLD] Google', '[BOLD] Analogy  [BOLD] SemEval']
In particular, the first category of methods are traditional static embeddings, where word vectors come from a lookup table. In the second category, static embeddings from dynamic word embedding models are used. In the last category, dynamic and static embeddings are integrated in the sense that context or center words representation are dynamically calculated via GCN or a BERT model for each sentence, but the target embeddings are static. Our model gives the best performance on 7 out of 8 datasets, achieving the best results on all the word similarity datasets (p-value < 0.01 using t-test). In particular, it outperforms the best performing baselines by a large margin on WS353, WS353R and Google datasets, obtaining 6.5%, 8.3%, and 4.2% improvement, respectively.