Learning to Summarize Passages: Mining Passage-Summary Pairs from Wikipedia Revision Histories
Table 4: Rouge evaluation results on Psg2Sum of various summarization models. The scores with 95% confidence interval are given by the official Rouge script. The best results are in bold.
['[BOLD] Models', 'Rouge-1 R', 'Rouge-1 P', 'Rouge-1 F1', 'Rouge-2 R', 'Rouge-2 P', 'Rouge-2 F1', 'Rouge-L R', 'Rouge-L P', 'Rouge-L F1']
We validate various models on the Psg2Sum dataset, including abstractive models (s2s), extractive models (LEAD1, TextRank, NN-SE) and mixed models (s2s+copy, PGN). We observe that extractive methods perform better in terms of Rouge Recall. For example, the NN-SE model achieves the best recall performance among all the baseline models, i.e., 43.76 Rouge-1 recall and 23.19 Rouge-2 recall. In the meanwhile, the abstractive models achieve better Rouge Precision scores. The s2s + copy model has the best precision performance in Rouge-1, -2 and -L. Surprisingly, we find that using coverage mechanism (PGN) leads to the precision drop but higher recall score (with longer output), although s2s+copy and PGN are statistically indistinguishable in terms of F1 score.