1 Introduction
Table 4: skipVAE and skip-sa-vae perform better than their counterparts (VAE, SAVAE) on the Yahoo corpus under all latent variable collapse metrics while achieving similar log-likelihoods. In particular, all latent dimensions are active when using skip-sa-vae. Perplexity (PPL) for the variational models is estimated by importance sampling of the log marginal likelihood with 200 samples from qϕ(z|\mathboldx).
['Model', 'Dim', 'PPL', '\\gls  [ITALIC] ELBO', 'KL', 'MI', 'AU']
We analyze the Yahoo Answers dataset from Yang et al. We first note that successfully training standard \glsplVAE for text with flexible autoregressive likelihoods such as LSTMs remains a difficult problem. We see that \glsVAE by itself experiences latent variables collapse. The \glsskipVAE is slightly better than \glsVAE at avoiding latent variable collapse for similar log likelihoods, although the KL is only marginally above zero and the model only has one active unit.