Do sequence-to-sequence VAEs learn global features of sentences?
Table 7: δ-VAE-style free bits with no KL annealing delivers the best downstream task performance and a KL closest to the desired rate. Ann.: 0: no annealing, 10: anneal for 10 epochs; FB: free bits type; F1(n): F1 score in the n data-regime; KL: rate obtained after training.
['FB', '[ITALIC] λ', 'Ann.', 'F1(5)', 'F1(50)', 'F1(500)', 'F1(5000)', 'F1(all)', 'KL(5)']
variant increase the actual rate that is reached at the end of the optimisation. Moreover, the increases are very unpredictable: we gain higher capacity due to using KL annealing when we are using the original free bits than when we are not. Therefore, we cannot hope to do comparisons with equal rates using KL annealing. The δ-VAE free bits variant without annealing reach the closest KL value to the desired target rate λ. In addition, the δ-VAE free bits without KL annealing consistently yield better downstream task performance. In summary, KL annealing is harmful when used with free bits and the δ-VAE free bits technique is superior to the original formulation. Therefore, all the experiments in the paper use the δ variant without annealing.