Spherical Latent Spaces for Stable Variational Autoencoders
Table 1: Development set KL and NLL values for two NVRNN models trained on the Penn Treebank with and without the annealing technique of bowman2016generating. The higher-capacity 3-layer model collapses when no annealing is used, and while annealing works to improve performance, it still does not perform as well as the 1-layer variant. By contrast, a variant of the 1-layer model with vMF gives an NLL value of 117 and a KL of 18.6, a stronger result relying more heavily on the latent variable.
['[EMPTY]', 'No annealing 3-layer', 'No annealing 1-layer', 'Sigmoid annealing 3-layer', 'Sigmoid annealing 1-layer']
We train an NVRNN model on the Penn Treebank with four different hyperparameter settings. We either use a 3-layer LSTM encoder or a 1-layer LSTM and use or do not use a sigmoid annealing schedule (increase the KL weight from 0 to 1 over the first 20 epochs). We observe the best performance using the 1-layer model with annealing. One might conclude from this table that the annealing trick has worked since both models achieve better performance when annealing is used. But in fact, a vMF-based model can do better than either (NLL of 117), and moreover, we have no way of knowing that a better annealing scheme might not achieve even higher performance after training. Furthermore, the higher-capacity 3-layer model can theoretically do anything the 1-layer model can, so its lower performance indicates that our training is derailed either by overfitting or getting stuck in a local optimum where the latent variable is unused.