“The Sum of Its Parts”:Joint Learning of Word and Phrase Representations with Autoencoders
Table 1: Word representations evaluation on both similarity and analogy tasks. Comparison of performance across all models with 100-dimensional word vector representations. For all models, a symmetric context window of ten words is used. Spearman rank correlation is reported on word similarity tasks. Accuracy is reported on word analogy tasks.
['[EMPTY]', '[BOLD] WS', '[BOLD] RG', '[BOLD] RW', '[BOLD] SYN.', '[BOLD] SEM.']
However, there is a significant performance boost between the low-rank approximation of X with a SVD and this same approximation with our joint model. This shows that combining a count-based model with a predictive-based approach helps for generating better word representations. Performance on word analogy tasks show that our joint model competes with others on the syntactic questions, but that it gives a lower accuracy on semantic questions. One possible explanation is that less common words are involved in semantic questions compared to syntactic questions. Among the four words that make a semantic question, one of them is, in average, the 34328th most frequent word in W, while it is the 20819th for a syntactic question. Compared to other methods which take the whole dictionary W as context dictionary, we consider only a small subset of it (D contains only the 10000 most frequent words of W).