Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization
Table 4: Comparison of different systems using the averaged ROUGE scores (1/2/L) with target summaries (R) and averaged oracle overlap ratios (SO, only for extractive systems). We calculate R between systems and selected summary sentences from each sub-aspect (R(P/D/I)) where each aspect uses the best algorithm: First, ConvexFall and NNearest. R(P/D/I) is rounded by the decimal point. - indicates the system has too few samples to train the neural systems. x indicates SO is not applicable because abstractive systems have no sentence indices. The best score for each corpora is shown in bold with different colors.
['[EMPTY]', '[EMPTY]', 'CNNDM R', 'CNNDM SO', 'CNNDM R(P/D/I)', 'XSum R', 'XSum SO', 'XSum R(P/D/I)', 'PeerRead R', 'PeerRead SO', 'PeerRead R(P/D/I)', 'PubMed R', 'PubMed SO', 'PubMed R(P/D/I)', 'Reddit R', 'Reddit SO', 'Reddit R(P/D/I)', 'AMI R', 'AMI SO', 'AMI R(P/D/I)', 'BookSum R', 'BookSum SO', 'BookSum R(P/D/I)', 'MScript R', 'MScript SO', 'MScript R(P/D/I)']
Neural extractive systems such as CL, SumRun and S2SExt outperform the others in general. LexRank is highly biased toward the position aspect. On the other hand, MMR is extremely biased to the importance aspect on XSum and Reddit. Interestingly, neural extractive systems are somewhat balanced compared to the others. Ensemble systems seem to have the three sub-aspects in balance, compared to the neural extractive systems. They also outperform the others (either ROUGE or SO) on five out of eight datasets.