Context encoders as a simple but powerful extension of word2vec
Table 1: Accuracy on the analogy task with mean and standard deviation computed using three random seeds when initializing the word2vec model. The best results for each category and corpus are in bold.
['[EMPTY]', 'text8 (10 iter) word2vec', 'text8 (10 iter) word2vec', 'text8 (10 iter) Context Encoder', 'text8 (10 iter) Context Encoder', '1-billion word2vec', '1-billion word2vec', '1-billion Context Encoder', '1-billion Context Encoder']
To capture some of the semantic relations between words (e.g. the first four task categories) it can be advantageous to use context encoders instead of word2vec. One reason for the ConEcs’ superior performance on some of the task categories, but not others, might be that the city and country names compared in the first four task categories only have a single sense (referring to the respective location), while the words asked for in other task categories can have multiple meanings. For example, “run” can be used as both a noun or a verb, additionally, in some contexts it refers to the sport activity while other times it is used in a more abstract sense, e.g. in the context of someone running for president. Therefore, the results in the other task categories might improve if the words’ context vectors are first clustered and then the ConEc embedding is generated by multiplying the word2vec embeddings with the average of only those context vectors corresponding to the word’s sense most appropriate for the task category.