Flexible End-to-End Dialogue System for Knowledge Grounded Conversation
Table 4: Human Evaluation on the MusicConvers dataset.
['Models', 'Grammar', 'Context Relevance', 'Correctness']
MusicConvers: We compute the mean score of each metric. For automatic evaluation, GenDS shows the best performance on BLEU and entity-accuracy, while GenDS-single achieves the highest entity-recall. Although GenDS does not overwhelm on S2SA in terms of BLEU, it improves entity accuracy and recall by 39% and 14% respectively. This indicates that GenDS can reply the message with more correct information. S2SA cannot respond with correct information, which is mainly due to the lack of grounding into external knowledge. Although GenQA and GenQAD can incorporate KB in response, their performance on entity-accuracy and entity-recall still cannot compete with GenDS. For GenQA, the entity generation probability is fixed during decoding. As a consequence, GenQA cannot generate different entities. For GenQAD, its update mechanism of entity generation probability is less effective than our dynamic knowledge enquirer. BLEU scores of GenQA and GenQAD are lowest among all models. We think that MusicConvers has no enough data for GenQA and GenQAD to learn reliable entity representations. This illustrate thet GenDS can achieve decent performance even on small dataset. GenDS achieves higher BLEU than GenDS-Single, which confirms the benefit of multi-task learning for improving the fluency. For human evaluation, the GenDS-single achieves the best performance in terms of grammar, context relevance and information correctness. Although S2SA can generate fluent responses, these responses contain little correct information and are less semantically relevant with the message. The performance of GenDS is slightly worse than GenDS-single. We infer that is due to the task 2, where GenDS tends to generate fluent response instead of correct information.