Discovering Discrete Latent Topics with Neural Variational Inference
Table 2: Perplexities of document models on the test datasets. The table compares the results for a fixed dimension latent variable, 50 or 200, achieved by our neural document models to Product of Experts LDA (prodLDA) (Srivastava & Sutton, 2016) and the Neural Variational Document Model (NVDM) (Miao et al., 2016).
['[BOLD] Finite Document Model', 'MXM 50', 'MXM 200', '20News 50', '20News 200', 'RCV1 50', 'RCV1 200']
According to our experimental results, the generalisation abilities of the GSM, GSB and RSB models are all improved by switching to an implicit topic distribution, and their performance is also significantly better than the NVDM and ProdLDA. We hypothesise that this effect is due to the models not needing to infer the topic-word assignments, which makes optimisation much easier. Interestingly, the RSB model performs better than the GSM and GSB on 20NewsGroups in both the 50 and 200 topic settings. This is possibly due to the fact that GSM and GSB apply linear transformations W1 and W2 to generate the hidden variable θ and breaking proportions η from a Gaussian draw, while the RSB applies recurrent neural networks to produce η in a sequence which induces dependencies in η and helps escape local minima. It is worth noting that the recurrent neural network uses more parameters than the other two models. If it is not necessary to model the explicit topic distribution over words, using an implicit topic distribution may lead to better generalisation.