Nonparametric Topic Modeling with Neural Inference
Table 2: The posterior distribution of α learned by iTM-VAE-HP on subsets of 20News dataset.
['#classes', '[ITALIC] γ1', '[ITALIC] γ2', 'E [ITALIC] q( [ITALIC] α)[ [ITALIC] α]']
We can see that, once the training is done, the variational posterior q(α|γ1,γ2) is very confident, and Eq(α|γ1,γ2)[α], the expectation of α given the variational posterior, is adjusted to the training set. For example, if the training set contains only 1 class of documents, Eq(α|γ1,γ2)[α] after training is 3.68, Whereas, when the training set consists of 10 classes of documents, Eq(α|γ1,γ2)[α] after training is 14.71. This indicates that iTM-VAE-HP can learn to adjust α to data, thus the number of discovered topics will adapt to data better. In contrast, for iTM-VAE-Prod (without the hyper-prior), when the decoder is strong, no matter how many classes the dataset contains, the number of topics will be constrained tightly due to the collapse-to-prior problem of AEVB, and KL-annealing and decoder regularizing tricks do not help much.