Modeling Online Discourse with Coupled Distributed Topics
Table 1: Perplexity of DDTM with and without coupling potentials (-cpl) vs. baselines trained at comment (cmt) or thread (thr) level across various numbers of topics and bits. For reference, a unigram model achieves 2644.
['Bits', '[BOLD] Perplexity (nats) 32', '[BOLD] Perplexity (nats) 64', '[BOLD] Perplexity (nats) 96', '[BOLD] Perplexity (nats) 128']
Setup: Due to the use of mean-field approximations for both the marginal energy and normalizer we lose any guarantees regarding the accuracy of our likelihood estimate (both approximations are lower bounds, and therefore their difference is neither a strict lower bound nor guaranteed to be unbiased). Note that since the marginal energy is intractable in our model, unlike a standard RBM, we must sample the joint - and not the marginal - intermediate distributions. This yields an unbiased estimate of the normalizer. The marginal energy must still be approximated via a lower bound, but given that AIS is unbiased and empirically low in variance, we can treat the overall estimate as a lower bound on likelihood for evaluation. Using 2000 intermediate distributions, and averaging over 20 runs, we evaluated per-word perplexity over a set of 50 unseen threads.