Towards Knowledge-Based Personalized Product Description Generation in E-commerce
Table 6. The gain from the proposed model KOBE over the baseline evaluated by pairwise human judgments.
['Model', 'Fluency', 'Diversity', 'Overall Quality']
Human Evaluation We conduct a human evaluation on a randomly sampled subset of the test set. Each instance contains an input product title, a description generated by the baseline and a description generated by KOBE. The selected instances are distributed to human annotators with no prior knowledge about the systems of the descriptions. Following Li et al. The score of fluency reflects how fluent the description is. The score of diversity reflects how diversified the description is and whether it contains much competitive content. The score of overall quality reflects whether the description is reasonable, or to say, whether it is consistent with world knowledge. The score range is from 1 to 5. The higher the score is, the more consistent with the criterion the description is. Consistent with the results of the automatic evaluation of diversity, KOBE outperforms the baseline by +0.06 in terms of diversity in human evaluation. However, the improvement in diversity does not sacrifice fluency. Instead, KOBE significantly outperforms the baseline by +0.17 in terms of fluency. As to the overall quality, KOBE still demonstrates a clear advantage over the baseline by +0.13, showing that our model is able to generate more reasonable descriptions with the introduction of external knowledge.