Dissecting Contextual Word Embeddings: Architecture and Representation
Table 3: Accuracy (%) for word vector analogies. In addition to the 300 dimension 840B GloVe vectors, the table contains results from a character n-gram hash and the context insensitive word embedding layer (xk) from the biLMs.
['[BOLD] Representation', '[BOLD] Syntactic', '[BOLD] Semantic']
The word analogy task introduced in Mikolov et al. Here, we use them to compare the word embedding layer from the biLMs to word vectors. The task has two types of analogies: syntactic with examples such as “bird:birds :: goat:goats”, and semantic with examples such as “Athens:Greece :: Oslo:Norway”. Traditional word vectors score highly on both sections. To further highlight this distinction, we also computed a purely orthographically based word vector by hashing all character 1, 2, and 3-grams in a word into a sparse 300 dimensional vector. As expected, vectors from this method had near zero accuracy in the semantic portion, but scored well on the syntactic portion, showing that most of these analogies can be answered with morphology alone. As a result, we conclude that the word representation layer in deep biLMs is only faithfully encoding morphology with little semantics.