Do sequence-to-sequence VAEs learn global features of sentences?
Table 4: Using BoW encoders, Uni decoders or PreLM pretraining, the representations learned by the VAEs are more predictive of the labels (sentiment or topic) of the documents.
['[EMPTY]', 'Enc.', '[ITALIC] r', 'Dec.', '[ITALIC] n/class Pre.', '5 F1± [ITALIC] σσinit', '50 F1± [ITALIC] σσinit', '500 F1± [ITALIC] σσinit', '5000 F1± [ITALIC] σσinit', 'All F1± [ITALIC] σσinit']
The proposed variants are either on par or improve significantly over the baselines. In the large data-regime, BoW-max-LSTM and LSTM-avg-LSTM-PreLM perform best on average. In the small data-regime, the picture is more complex and it depends on the dataset. The exception is LSTM-last-Uni which is worse than the PreAE baselines and suffers from unstable training on AGnews (high variance).