Learning Word Representations with Hierarchical Sparse Coding
Table 2: Results on the syntactic and semantic analogies tasks with a bigger corpus (M=260).
['Task', 'CBOW', 'SG', 'forest']
We hypothesize that this is because performing well on these tasks requires training on a bigger corpus. We combine our WMT-2011 corpus with other news corpora and Wikipedia to obtain a corpus of 6.8 billion words. The size of the vocabulary of this corpus is 401,150. (M=52 does not perform as well, and M=520 is computationally expensive). All models benefit significantly from a bigger corpus, and the performance levels are now comparable with previous work. On the syntactic analogies task, forest is the best model. On the semantic analogies task, SG outperformed forest, and they both are better than CBOW.