Document Neural Autoregressive Distribution Estimation
Table 1: Test perplexity per word for models with 50 topics. The results for LDA and Replicated Softmax were taken from Salakhutdinov and Hinton (2009).
['Dataset', 'LDA', 'Replicated Softmax', 'fDARN', 'DocNADE', 'DeepDN (1layer)', 'DeepDN (2layer)', 'DeepDN (3layer)']
Theere we compare 5 different models: the Latent Dirichlet Allocation (LDA) (DeepDN in the table). Each model uses 50 latent topics. For the experiments with DeepDocNADE, we provide the performance when using 1, 2, and 3 hidden layers. The hyper-parameters of Adam were selected on the validation set. Crucially, following Uria et al. Specifically, the perplexity with M ensembles becomes as follows: exp(−1T∑t1|vt|log(1M∑mp(v(t,m)))), (26) where T is the total number of examples , M is the number of ensembles (word orderings) and v(t,m) denotes the mth word ordering for the tth documnet. For the 20 Newsgroups dataset, adding more hidden layers to DocDocNADE fails to provide further improvements. We hypothesize that the relatively small size of this dataset makes it hard to successfully train a deep model. However, the opposite is observed on the RCV1-V2 dataset, which is more than an order of magnitude larger than 20 Newsgroups. In this case, DeepDocNADE outperforms fDARN and DocNADE, with a relative perplexity reduction of 20%, 24% and 26% with respectively 1,2 and 3 hidden layers.