A Multi-Turn Emotionally Engaging Dialog Model
Table 2. Perplexity and average BLEU scores achieved by the models. Avg. BLEU: average of BLEU-1, -2, -3, and -4. Validation set 1 comes from the Cornell dataset, and validation set 2 comes from the DailyDialog dataset.
['(lr)2-4 (lr)5-7', 'Perplexity Validation Set 1', 'Perplexity Validation Set 2', 'Perplexity Test Set', 'Avg.\xa0BLEU Validation Set 1', 'Avg.\xa0BLEU Validation Set 2', 'Avg.\xa0BLEU Test Set']
As shown in the table, MEED achieves the lowest perplexity and the highest BLEU score on all three sets. We conducted t-test on the perplexity obtained, and results show significant improvements of MEED over S2S and HRAN on the two validation sets (with p-value <0.05). The comparison between perplexity scores and human evaluation results further confirms the fact that in the context of dialog response generation, perplexity does not align with human judgement. However, for all of the three criteria in human evaluation, HRAN actually outperforms S2S. Based on this, we conclude that perplexity alone is not enough for evaluating a dialog system.