Learning Meta-Embeddings by Using Ensembles of Embedding Sets
Table 3: Comparison of effectiveness of four methods for learning OOV embeddings. RND: random initialization. AVG: average of embeddings of known words. ml: MutualLearning. RW(21) means there are still 21 OOVs for the vocabulary union.
['[EMPTY]', '[EMPTY]', 'RW(21) RND', 'RW(21) AVG', 'RW(21) ml', 'RW(21) 1toN+', 'semantic RND', 'semantic AVG', 'semantic ml', 'semantic 1toN+', 'syntactic RND', 'syntactic AVG', 'syntactic ml', 'syntactic 1toN+', 'total RND', 'total AVG', 'total ml', 'total 1toN+']
System comparison of learning OOV embeddings. As both word2vec and GloVe have full coverage on analogy, we do not rereport them in this table. For each embedding set, we can compute the representation of an OOV (i) as a randomly initialized vector (RND); (ii) as the average of embeddings of all known words (AVG); (iii) by MutualLearning (ml) and (iv) by 1toN+. 1toN+ learns OOV embeddings for individual embedding sets and meta-embeddings simultaneously, and it would not make sense to replace these OOV embeddings computed by 1toN+ with embeddings computed by “RND/AVG/ml”. Hence, we do not report “RND/AVG/ml” results for 1toN+. (i) MutualLearning helps much if an embedding set has lots of OOVs in certain task; e.g., MutualLearning is much better than AVG and RND on RW, and outperforms RND considerably for CONC, SVD and 1toN on analogy. However, it cannot make big difference for HLBL/CW on analogy, probably because these two embedding sets have much fewer OOVs, in which case AVG and RND work well enough. (ii) AVG produces bad results for CONC, SVD and 1toN on analogy, especially in the syntactic subtask. We notice that those systems have large numbers of OOVs in word analogy task. If for analogy “a is to b as c is to d”, all four of a,b,c,d are OOVs, then they are represented with the same average vector. Hence, similarity between b−a+c and each OOV is 1.0. In this case, it is almost impossible to predict the correct answer d. (iii) MutualLearning learns very effective embeddings for OOVs. CONC-ml, 1toN-ml and SVD-ml all get better results than word2vec and GloVe on analogy (e.g., for semantic analogy: 88.1, 87.3, 88.2 vs. 81.4 for GloVe). Considering further their bigger vocabulary, these ensemble methods are very strong representation learning algorithms. (iv) The performance of 1toN+ for learning embeddings for OOVs is competitive with MutualLearning. For HLBL/Huang/CW, 1toN+ performs slightly better than MutualLearning in all four metrics. Comparing 1toN-ml with 1toN+, 1toN+ is better than “ml” on RW and semantic task, while performing worse on syntactic task.