Embedding Word Similarity with Neural Machine Translation
Table 1: NMT embeddings (RNNenc and RNNsearch) clearly outperform alternative embedding-learning architectures on tasks that require modelling similarity (below the dashed line), but not on tasks that reflect relatedness. Bilingual embedding spaces learned without the translation objective are somewhere between these two extremes.
['[EMPTY]', '[EMPTY]', 'Monolingual models  [BOLD] Skipgram', 'Monolingual models  [BOLD] Glove', 'Monolingual models  [BOLD] CW', 'Biling. models  [BOLD] FD', 'Biling. models  [BOLD] BiCVM', 'NMT models  [BOLD] RNNenc', 'NMT models  [BOLD] RNNsearch']
NMT outperform monolingual embeddings, and, to a lesser extent, the other bilingually trained embeddings, on SimLex-999. Given the aforementioned differences between the evaluations, this suggests that bilingually-trained embeddings, and NMT based embeddings in particular, better capture similarity, whereas monolingual embedding spaces are orientated more towards relatedness. NMT embeddings outperform other bilingually-trained embeddings which in turn outperform monolingual models. The difference is particularly striking on SimLex-Assoc-333, which suggests that the ability to discern similarity from relatedness (when relatedness is high) is perhaps the most clear distinction between the bilingual spaces and those of monolingual models.