Spherical Latent Spaces for Stable Variational Autoencoders
Table 6: Average cosine similarity when trying to reconstruct the latent code μ from the bag of words and vice versa. In vMF, the latent code contains more information beyond the bag of words, as shown by the lower cosine similarity when predicting BoW →μ (0.57). When the latent code is learned in a model conditioned on the bag of words (right column), it predicts the bag of words much less well, indicating that the model successfully learns orthogonal information.
['Model Setting', 'NVRNN  [ITALIC] μ→ BoW', 'NVRNN BoW → [ITALIC] μ', 'NVRNN-BoW  [ITALIC] μ→ BoW']
For vMF, μ can reconstruct the bag-of-words more accurately than the bag-of-words can reconstruct μ, indicating that the latent code in vMF captures more information beyond the bag of words. We repeat this experiment in a separate NVRNN model where the decoder can explicitly condition on the BoW vector described above. Our model, v-VAE, achieves a lower cosine similarity than G-VAE (0.23 vs. 0.32), indicating that it capturing less redundant information and using the latent space to more efficiently model other properties of the data.