1 Introduction
Table 2: Performance of skipVAE vs VAE on MNIST (Top) and Omniglot (Bottom) as the complexity of the generative model increases. The number of latent dimension is fixed at 20. Skip-VAE outperforms VAE on all collapse metrics while achieving similar ELBO values, and the difference widens as layers increase.
['[EMPTY]', 'Layers', 'ELBO VAE', 'ELBO skipVAE', 'KL VAE', 'KL skipVAE', 'MI VAE', 'MI skipVAE', 'AU VAE', 'AU skipVAE']
We use a model with 20-dimensional latent variables. For \glsVAE, as the generative model becomes more expressive the model becomes less reliant on z. We see this in the poor performance on the collapse metrics. The \glsskipVAE mitigates this issue and performs better on all latent-variable collapse metrics. Note the \glsELBO is similar for both models. These results indicate that the family of generative skip models has a strong inductive bias to share more mutual information between the observation and the latent variable.