Question Answering as an Automatic Evaluation Metric for News Article Summarization
Table 2: Correlation matrix of ROUGE and APES.
['[EMPTY]', 'R-1', 'R-2', 'R-L', 'R-SU', 'APES']
While ROUGE baselines were beaten only by a very small number of suggested metrics in the original AESOP task, we find that APES shows better correlation than the popular R-1, R-2 and R-L, and the strong R-SU. Although showing statistical significance for our hypothesis is difficult because of the small dataset size, we claim APES gives an additional value comparing to ROUGE: In contrast, APES is not correlated with ROUGE metrics to the same extent (around 0.6). The above suggests that APES offers additional information regarding the text in a manner that ROUGE does not. For this reason, we believe APES complements ROUGE.