Cyclical Annealing Schedule:A Simple Approach to Mitigating KL Vanishing
Table 5: Comparison on dialog response generation. BLEU (B) scores 1-4 are used for evaluation. Monotonic (M) and Cyclical (C) schedules are tested on two models.
['[BOLD] Model  [BOLD] Schedule', 'CVAE  [BOLD] M', 'CVAE  [BOLD] C', 'CVAE+BoW  [BOLD] M', 'CVAE+BoW  [BOLD] C']
A4SS1SSS0Px2 Results The cyclical schedule outperforms the monotonic schedule in both settings. Under similar ELBO results, the cyclical schedule provide lower reconstruction errors, higher KL values, and higher BLEU values than the monotonic schedule. Interestingly, the monotonic schedule tends to overfit, while the cyclical schedule does not, particularly on reconstruction errors. It means the monotonic schedule can learn better latent codes for VAEs, thus preventing overfitting.