KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation
Table 6: Manual evaluation. The best results (t-test, p-value < 0.005) are in bold. Between two generative models, the significantly better results are italic underlined (t-test, p-value < 0.005) or underlined (t-test, p-value < 0.05). κ is the Fleiss’ kappa value. “+ know” means the models enhanced by knowledge information.
['[BOLD] Model', '[BOLD] Fluency', '[BOLD] Coherence']
4.4.3 Results As can be seen, knowledge-aware BERT outperforms other models significantly in both metrics in all the three domains, which agrees with the results of automatic evaluation. The Fluency is 2.00 because the retrieved responses are all human-written sentences. The Fluency scores of both generation-based models are close to 2.00 (in the music domain, the Fluency of HRED is 1.90), showing that the generated responses are fluent and grammatical. The Coherence scores of both HRED and knowledge-aware HRED are higher than 1.00 but still have a huge gap to 2.00, indicating that the generated responses are relevant to the context but not coherent to knowledge information in most cases. After incorporating the knowledge information into HRED, the Coherence score is improved significantly in all the three domains, as the knowledge information is more expressed in the generated responses.