Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
Table 1: Performance of each model on automatic measures. The highest score in each row is in bold. Note that our BLEU scores are normalized to [0,1].
['Metrics', 'Baseline', 'CVAE', 'kgCVAE']
Automatically evaluating an open-domain generative dialog model is an open research challenge Liu et al. Following our one-to-many hypothesis, we propose the following metrics. We assume that for a given dialog context c, there exist Mc reference responses rj, j∈[1,Mc]. Meanwhile a model can generate N hypothesis responses hi, i∈[1,N]. The generalized response-level precision/recall for a given dialog context is: precision(c) =∑Ni=1maxj∈[1,Mc]d(rj,hi)N recall(c) =∑Mcj=1maxi∈[1,N]d(rj,hi))Mc where d(rj,hi) is a distance function which lies between 0 to 1 and measures the similarities between rj and hi. Papineni et al. We use BLEU-1 to 4 as our lexical similarity metric and normalize the score to 0 to 1 scale. Cosine Distance of Bag-of-word Embedding: a simple method to obtain sentence embeddings is to take the average or extrema of all the word embeddings in the sentences Forgues et al. Adi et al. The d(rj,hi) is the cosine distance of the two embedding vectors. The score is normalized to [0, 1]. We set d(rj,hi)=1 if rj and hi have the same dialog acts, otherwise d(rj,hi)=0. One challenge of using the above metrics is that there is only one, rather than multiple reference responses/contexts. This impacts reliability of our measures. Inspired by Sordoni et al. Then the 10 candidate references are filtered by two experts, which serve as the ground truth to train the reference response classifier. The result is 6.69 extra references in average per context. The average number of distinct reference dialog acts is 4.2.