Learning to Select Knowledge for Response Generation in Dialog Systems
Table 2: Automatic and Human Evaluation on Persona-chat and Wizard-of-Wikipedia
['Dataset', 'Model', 'Automatic Evaluation BLEU-1/2/3', 'Automatic Evaluation Distinct-1/2', 'Automatic Evaluation Knowledge R/P/F1', 'Human Evaluation']
Among them, BLEU-1/2/3 and Distinct-1/2 are two widely used metrics for evaluating the quality and diversity of generated responses. Specifically, given the set of non-stopwords in Y and in the knowledge collection {Ki}Ni=1, denoted by WY and WK, we define Knowledge R(ecall) and Knowledge P(recision) to be |WY∩WK|/|WK| and |WY∩WK|/|WY| and Knowledge F1 = 2⋅Recall⋅PrecisionRecall+% Precision. Specifically, compared with Seq2Seq, incorporating knowledge is shown to be helpful in generating diverse responses. For example, Distinct-1/2 on Persona-chat is increased from 0.026/0.074 (Seq2Seq) to 0.048/0.126 (PostKS(concat)) , meaning that the diversity is greatly improved by augmenting with knowledge. Besides, when comparing with existing knowledge-grounded baselines, our models demonstrate their ability on incorporating appropriate knowledge in response generation. In particular, comparing PostKS(fusion) against MemNet(soft) on Persona-chat (they are soft knowledge-grounded models except that we use both prior and posterior information to facilitate knowledge selection), we achieve higher BLEU and Distinct scores. This is because that the posterior information is better utilized in our models to provide effective guidance on obtaining appropriate knowledge, resulting in responses with better quality. Compared with knowledge selection on Persona-chat, selecting appropriate knowledge on Wizard-of-Wikipedia is more challenging due to a larger knowledge collection size. Nevertheless, our models perform consistently better than most baselines. For example, PostKS(fusion) has higher knowledge R/P/F1 compared with all MemNet based models on Wizard-of-Wikipedia, indicating that it can not only select appropriate knowledge, but also ensure that knowledge is better incorporated in the response generated. Finally, we observe that PostKS(fusion) performs slightly better than PostKS(concat) in most cases. This verifies that soft knowledge incorporation is a better way of introducing knowledge to response generation since it allows for more flexible knowledge integration and less sensitivity to noise. In human evaluation, three annotators were recruited to rate the overall quality of the responses generated by each model. The rating ranges from 0 to 2, where 0 means that the response is completely irrelevant, 1 means that the response is acceptable but not very informative, and 2 means that the response is natural, relevant and informative. We randomly sampled 300 responses for each model on each dataset, resulting in 4,500 responses in total for human annotation. According to the result, both of our models, PostKS(concat) and PostKS(fusion), are remarkably better than all existing baselines in terms of human rating, demonstrating the effectiveness of our novel knowledge selection mechanism.