Facet-Aware Evaluation for Extractive Summarization
Table 5: ROUGE-1 F1 of extractive and abstractive methods on noisy (N), low abstraction (L), high abstraction (H), and high quality (L + H) samples.
['[EMPTY]', '[BOLD] Method', '[BOLD] N', '[BOLD] L', '[BOLD] H', '[BOLD] L + H']
Abstractive methods, however, also exhibit a huge performance gap between low and high abstraction samples, which suggests that existing abstractive methods achieve decent overall performance mainly by extraction rather than abstraction, i.e., performing well on low abstraction samples of CNN/Daily Mail. We also found that all the compared methods perform much worse on the documents with “noisy” reference summaries, implying that the randomness in the reference summaries might introduce noise to both model training and evaluation. Note that although the sample size is relatively small, we observe consistent results when analyzing different subsets of the data.