A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
Table 2: Results on PTB test with encoder pretraining.
['[BOLD] Method', '[BOLD] PPL↓', '[BOLD] Recon↓', '[BOLD] AU↑', '[BOLD] KL', '[BOLD] -ELBO']
Based on the observations above we hypothesize that VAEs might benefit from initialization with an non-collapsed encoder, trained via an AE objective. Intuitively, if the encoder is providing useful information from the beginning of training, the decoder is more likely to make use of the latent code. Even with encoder pretraining, we see that posterior collapse occurs immediately after beginning to update both encoder and decoder using the full ELBO objective. This indicates that the gradients of ELBO point towards a collapsed local optimum, even with biased initialization. When pretraining is combined with annealing, PPL improves substantially. However, the pretraining and anneal combination only has 2 active units and has small KL value – the latent representation is likely unsatisfactory. We speculate that this is because the annealing schedule eventually returns to the full ELBO objective which guides learning towards a (nearly) collapsed latent space. In the next section, we present an alternate approach using the KL thresholding / free bits method.