Do sequence-to-sequence VAEs learn global features of sentences?
Table 3: When the KL collapses, the performances of downstream task classifiers trained on the mean μ vs on samples z∼N(μ,Iσ2) are very different, especially for pretrained models. z does not contain any information while μ is very predictive of the label.
['PreAE', 'F1  [ITALIC] z', 'F1  [ITALIC] μ', 'KL']
We train the original VAE architecture on the Yelp dataset, both with and without the PreAE, using the original ELBo objective (λ=0). As expected, the KL term collapses to 0. Then, we train a classifier using the procedure explained above using 5000 examples per class. We expect that its performance will be close to random chance, regardless of whether samples or the mean parameter are used as inputs. Using samples, we do get random chance predictions from the classifiers, whereas using means, the performance is remarkably high (as high as 81.5 of F1 using pretraining). The reason is that the KL term never completely collapses to 0. Therefore, μ can be almost zero while still encoding a lot of information about its inputs. However, when the KL term is close to 0, the variance of the samples is close to 1, so no information is transmitted to the decoder. This tendency is exacerbated with the PreAE runs, for which the means encode remnants of the pretraining phase.