Knowledge Aware Conversation Generation with Explainable Reasoning over Augmented Graphs
Table 3: Results of human evaluations on the two datasets. AKGCM (or AKGCM-5) outperforms all the baselines significantly (sign test, p-value < 0.05) in terms of the two metrics.
['[EMPTY]', 'EMNLP dialog dataset Appr.', 'EMNLP dialog dataset Infor.', 'ICLR dialog dataset Appr.', 'ICLR dialog dataset Infor.']
Results: We see that our model outperforms all the baselines significantly (sign test, p-value < 0.05) in terms of the two metrics on the two datasets. Furthermore, our model can beat the strongest baseline, BiDAF. It demonstrates the effectiveness of our graph reasoning mechanism that can use global graph structure information and exploit long text information. Our data analysis shows that both Seq2Seq and HRED tend to generate safe responses starting with “my favorite character is” or “I think it is”. Both Memnet and TMemnet can generate informative responses. But the knowledge in their responses tends to be incorrect, which is a serious problem for knowledge aware conversation generation. Our results show that GTTP and BiDAF are very strong baselines. It indicates that the attention mechanism (from machine reading) for knowledge selection and the copy mechanism can bring benefits for knowledge aware conversation generation. Although CCM have the mechanisms mentioned above, this model is good at dealing with structured triples rather than long texts. It may explain the inferior performance of CCM in our problem setting.