Automatic Article Commenting: the Task and Dataset
Table 4: Correlation between metrics and human judgments on comments. “Human” represents the results from randomly dividing human judgments into two groups. All values are with p-value <0.01.
['Metric', '[BOLD] Spearman', '[BOLD] Pearson']
The METEOR family correlates best with human judgments, and the enhanced weighted metrics improve over their vanilla versions in most cases (including BLEU-2/3 as in the supplements). E.g., the Pearson of METEOR is substantially improved from 0.51 to 0.57, and the Spearman of ROUGE_L from 0.19 to 0.26. Compared to W-METEOR, METEOR deviates from the regression line more frequently, esp. by assigning unexpectedly high scores to comments with low human grades. For instance, our proposed weighted metrics substantially improve the Pearson correlation of METEOR from 0.51 to 0.57, and the Spearman correlation of ROUGE_L from 0.19 to 0.26.