Multi-turn Dialogue Response Generation in an Adversarial Learning Framework
Table 1: Generator Performance Evaluation
['Model', 'Teacher Forcing Perplexity', 'Teacher Forcing − [ITALIC] logD( [ITALIC] G(.))', 'Autoregression BLEU-2', 'Autoregression ROUGE-2', 'Autoregression DISTINCT-1/2', 'Autoregression NASL', 'Human Evaluation']
Perplexity: The table indicates that both variants of the hredGAN model perform better than the HRED and VHRED models in terms of the perplexity measure. However, using the adversarial loss criterion (Eq. Note that, for this experiment, we run all models in teacher forcing mode. Quantitative Generator Performance : We run autoregressive inference for all the models (using optimum α values for hredGAN models and selecting the best of L=64 responses using a discriminator) with dialogue contexts from a unique test set. Distinct(1/2) largely agrees with the perplexity score. Most scores, similar to the perplexity, indicate that hredGAN models perform better than (V)HRED on both datasets. However, on the UDC ROUGE and MTC BLEU, VHRED scores slightly better than hredGAN_u but still worse than hredGAN_w. Qualitative Generator Performance: The human evaluation agrees largely with the automatic evaluation. hredGAN_w performs best on both datasets although the gap is more on the MTC than on the UTC. This implies that the improvement of HRED with adversarial generation is better than with variational generation (VHRED). While other models produce short and generic utterances, hredGAN_w mostly yields informative responses. We see similar behavior across other samples. We also note that although hredGAN_u’s responses are the longest on Ubuntu (in line with the NASL score), the responses are less informative compared to hredGAN_w resulting in a lower human evaluation score. We reckon this might be due to a mismatch between utterance-level noise and word-level discrimination or lack of capacity to capture the data distribution using single noise distribution. We hope to investigate this further in the future.