Discovering Discrete Latent Topics with Neural Variational Inference
Table 1: Perplexities of the topic models on the test datasets. The upper section of the table lists the results for finite neural topic models, with 50 or 200 topics, on the MXM, 20NewsGroups and RCV1 datasets. We compare our neural topic models with the Gaussian Softmax (GSM), Gaussian Stick Breaking (GSB) and Recurrent Stick Breaking (RSB) constructions to the online variational LDA (onlineLDA) (Hoffman et al., 2010) and neural variational inference LDA (NVLDA) (Srivastava & Sutton, 2016) models. The lower section shows the results for the unbounded topic models, including our truncation-free RSB (RSB-TF) and the online HDP topic model (Wang et al., 2011).
['[BOLD] Finite Topic Model', 'MXM 50', 'MXM 200', '20News 50', '20News 200', 'RCV1 50', 'RCV1 200']
We use Perplexity as the main metric for assessing the generalisation ability of our generative models. Here we use the variational lower bound to estimate the document perplexity: exp(−1D∑Dd1Ndlogp(d)) following Miao et al. Amongst the finite topic models, the Gaussian softmax construction (GSM) achieves the lowest perplexity in most cases, while all of the GSM, GSB and RSB models are significantly better than the benchmark LDA and NVLDA models. Here we see that the recurrent neural topic model performs significantly better than the HDP topic model on perplexity.