Do sequence-to-sequence VAEs learn global features of sentences?
Table 6: Our variants reconstruct the inputs with 1) higher agreement with the ground-truth, 2) less memorization of the 1st word and the length, 3) with a negligible loss in likelihood. The best score and scores within one std are bolded.
['[EMPTY]', 'Enc.', '[ITALIC] r', 'Pre.', 'Agreement', '1st (%)', 'Len (%)', 'NLL']
There is a close correspondence between agreement and performance on the SSL tasks in the large data-regime. Our variants have higher agreement than the baseline, especially on Amazon and Yahoo datasets where, as we have seen before, the memorization of the first words is an especially bad inductive bias. Note that on these datasets, the agreements are consistently lower than the downstream task performance classification, which shows that reconstructing a sentence with the same label as the source sentence is harder than predicting the label using a classifier. Apart from that, the agreement does not tell us much more than the SSL results.