Improving Conversational Recommender Systems viaKnowledge Graph based Semantic Fusion
Table 3. Automatic evaluation results on the conversation task. We abbreviate Distinct-2,3,4 as Dist-2,3,4. Numbers marked with * indicate that the improvement is statistically significant compared with the best baseline (t-test with p-value <0.05).
['Models', 'Dist-2', 'Dist-3', 'Dist-4', 'Item Ratio']
Automatic Evaluation First, ReDial performs better than Transformer in Distinct-2/3/4, since it utilizes a pre-training RNN model to encode history utterances. While, Transformer performs better with the metric of Item Ratio. A possible reason is that Transformer architecture adopts the self-attention mechanism for capturing temporal pairwise interaction, which is more suitable to model the relations between words and items than RNN and CNN. Second, among the three baselines, KBRD generates the most diverse responses and achieves the highest item ratio, i.e., containing more mentions of items in the generated text. This model utilizes KG information to promote the predictive probability of entities and items. Compared with these baselines, our KGSF model is consistently better in all evaluation metrics. KGSF has utilized the KG information in two major steps, namely the knowledge-enhanced Transformer decoder and the copy mechanism, which enhances the informativeness of the generated text.