Attention Word Embedding
Table 1: Spearman correlation on word similarity datasets for AWE, CBOW, Skip-Gram, and GloVe.
['[BOLD] Model', '[BOLD] MEN', '[BOLD] WS353', '[BOLD] WS353R', '[BOLD] WS353S', '[BOLD] SimLex999', '[BOLD] RW(RareWords)', '[BOLD] RG65', '[BOLD] MTurk']
We use spearman correlation between cosine similarity and human-annotated relatedness scores as the metric to measure the performance. AWE performs significantly better on all of the datasets except SimLex999. Both algorithms incorporate additional subword information into their architecture. FastText uses character n-grams, while AWE-S uses the lemma forms of the words. AWE wins against FastText on six out of the eight datasets. A logistic regression classifier is trained to classify sentences using pre-trained word embeddings. For these initialization applications, AWE provides improvement in performance as compared to other models over all datasets except SICK-E. Subwords play a critical function in the robust learning of infrequent words. For instance, the word happiest may not occur as frequently as the word happy, but the model can learn more about the word happiest if it were to supplement it with the information of the word happy as well. This is the reason why fastText and AWE-S perform significantly better than other embedding methods on the RareWords dataset FastText relates the two words happy and happiest through the n-gram ‘happ’. On the other hand, AWE-S relates the word happiest to happy through its verb lemma form, i.e, ‘happy’.