Cyclical Annealing Schedule:A Simple Approach to Mitigating KL Vanishing
Table 3: Comparison on dialog response generation. Reconstruction perplexity (Rec-P) and BLEU (B) scores are used for evaluation.
['[BOLD] Model  [BOLD] Schedule', 'CVAE  [BOLD] M', 'CVAE  [BOLD] C', 'CVAE+BoW  [BOLD] M', 'CVAE+BoW  [BOLD] C']
(i) Smoothed Sentence-level BLEU Chen and Cherry : BLEU is a popular metric that measures the geometric mean of modified n-gram precision with a length penalty. We use BLEU-1 to 4 as our lexical similarity metric and normalize the score to 0 to 1 scale. (ii) Cosine Distance of Bag-of-word Embedding Liu et al. We used Glove embedding and denote the average method as A−bow and extreme method as E−bow. The score is normalized to [0,1]. Higher values indicate more plausible responses. The BoW indeed reduces the KL vanishing issue, as indicated by the increased KL and decreased reconstruction perplexity. When applying the proposed cyclical schedule to CVAE, we also see a reduced KL vanishing issue. Interestingly, it also yields the highest BLEU scores. This suggests that the cyclical schedule can generate dialog responses of higher fidelity with lower cost, as the auxiliary BoW loss is not necessary.