A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
Table 1: Results on PTB test set for various baselines.
['[BOLD] Method', '[BOLD] PPL↓', '[BOLD] Recon↓', '[BOLD] AU↑', '[BOLD] KL', '[BOLD] -ELBO']
the systems with relatively good reconstruction (FBP, λ=7) or higher AU (FB baselines) have suboptimal PPL, and the best PPL is achieved with sacrifice of Recon and AU. Note that good PPL indicates good LM, but good recon and AU indicates good representation learning. Without both, we do not really have a strong probabilistic model of language that captures latent factors. First, reconstruction loss for an AE is substantially better than all VAE methods, which is intuitive since reconstruction is the only goal of training an AE. Second, models with high ELBO do not necessarily have good PPL (e.g. VAE+anneal); ELBO is not an ideal surrogate for evaluating language modeling performance.