1 Introduction
Table 4: Mean reciprocal rank of held-out context words. SG = skip-gram, TM = topic model, MM = mixed membership. Bold indicates statistically significant improvement versus SG.
['Dataset', 'Frequency baseline', 'Google +context', 'SG', 'SG +context', 'MMSG prior', 'MMSG posterior', 'SGTM', 'SGTM +context', 'MMSGTM prior', 'MMSGTM posterior']
I compared to a word frequency baseline, the skip-gram (SG), and Tomas Mikolov/Google’s vectors trained on Google News, N≈100 billion, via CBOW. Simulated annealing was performed for 1,000 iterations, NCE was performed for 1 million minibatches of size 128, and 128-dimensional embeddings were used (300 for Google). I used K=2,000 for NIPS, K=500 for state of the Union, and K=100 for the two smaller datasets. We can identify several noteworthy findings. First, the generic big data vectors (Google+context) were outperformed by the skip-gram on 3 out of 4 datasets (and by the skip-gram topic model on the other), by a large margin, indicating that domain-specific embeddings are often important. Second, the mixed membership models, using posterior inference, beat or matched their naive Bayes counterparts, for both the word embedding models and the topic models. As hypothesized, posterior inference on zi at test time was important for good performance. Finally, the topic models beat their corresponding word embedding models at prediction. I therefore recommend the use of our MMSG topic model variant for predictive language modeling in the small data regime.