Variational Transformers for Diverse Response Generation
Table 1: Results of Variational Transformer compared to baselines on automatic and human evaluations.
['[BOLD] MojiTalk  [BOLD] Model', '[BOLD] MojiTalk  [BOLD] PPL', '[BOLD] MojiTalk  [BOLD] KLD', '[BOLD] MojiTalk  [BOLD] Diversity', '[BOLD] MojiTalk  [BOLD] Diversity', '[BOLD] MojiTalk  [BOLD] Diversity', '[BOLD] MojiTalk  [BOLD] Embeddings Similarity', '[BOLD] MojiTalk  [BOLD] Embeddings Similarity', '[BOLD] MojiTalk  [BOLD] Human Evaluation', '[BOLD] MojiTalk  [BOLD] Human Evaluation']
Transformer-based models have significantly lower perplexity compared to RNN-based models which indicate that the global receptive field performed by multi-head self-attention boost the modeling capacity. However, deterministic Seq2Seq and Transformer models tends to generate generic responses which leads to a low diversity score. Meanwhile incorporating a stochastic latent variable into both models (CVAE and GVT) promote more diverse generation results and boost the diversity scores such as Dist-1, Dist-2, and Dist-3. On the other hand, SVT achieves the highest score in terms of two semantic relevance-oriented metrics such as EMBFT and EMBBERT in MojiTalk dataset, while in the combined dataset of Persona and ED, we observe performance drop of SVT compare to other models. This is because both Persona and ED are well designed and have lower entropy than MojiTalk which collected from Twitter. We hypothesize that the sequential latent variables have no advantage in term of similarity to single, fixed ”gold response” when model low entropy response. Indeed, in open domain dialogue response generation, automatic metric is not always aligned with the human judgement Liu et al.