Do sequence-to-sequence VAEs learn global features of sentences?
Table 1: Datasets characteristics. |Y|: number of different labels. H[Y]: entropy of labels. NLL: mean negative log-likelihood of LSTM baseline models (std. over 3 runs). Splits size: train/valid/test sizes in thousands.
['Dataset', 'Splits size', 'Label', '|Y|', '[ITALIC] H[ [ITALIC] Y]', 'NLL']
We train VAEs on four small versions of AGNews, Amazon, Yahoo and Yelp from Zhang et al. Each document is written in English and consists of one or several sentences. Each document is manually labeled according to its main topic or the sentiment it expresses, and the labels are close to uniformly balanced over all the dataset. For faster training, we use smaller datasets. Indeed, λ=2 nats is enough to store the labels of all datasets without any loss, except Yahoo which has an entropy of 2.3 whereas λ=8 nats suffices to capture much more information than needed to store the labels on all datasets. Moreover, these rates are chosen to be much smaller than the reconstruction loss of the baselines because of the technical difficulty of increasing the rate without degrading the log-likelihood explained above.