Read, Attend and Comment: A Deep Architecture for Automatic News Comment Generation
Table 4: Evaluation results on automatic metrics and human judgment. Human evaluation results are calculated by combining labels from the three judges. “Kappa” means Fleiss’ kappa. Numbers in bold mean that improvement over the best baseline is statistically significant.
['Dataset', 'Models', 'METEOR', 'W-METEOR', 'Rouge_L', 'W-Rouge_L', 'CIDEr', 'W-CIDEr', 'BLEU-1', 'W-BLEU-1', 'Human', 'Kappa']
On most automatic metrics, DeepCom outperforms baseline methods, and the improvement is statistically significant (t-test with p-value <0.01). The improvement on BLEU-1 and W-BLEU-1 is much bigger than that on other metrics. This is because BLEU-1 only measures the proportion of matched unigrams out of the total number of unigrams in the generated comments. In human evaluation, although the absolute numbers are different from those reported in Qin et al. In human evaluation, the values of Fleiss’ kappa over all models are more than 0.6, indicating substantial agreement among the annotators. Although built in a complicated structure, GANN does not bring much improvement over other baseline methods, which demonstrates that only using news titles is not enough in comment generation. IR-TC and Att-TC represent the best retrieval model and the best generation model among the baselines on both datasets, implying that news bodies, even used in a simple way, can provide useful information to comment generation.