Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Table 2: Corpus bias towards extractive methods in the CNN, DailyMail, NY Times, and XSum datasets. We show the proportion of novel n-grams in gold summaries. We also report ROUGE scores for the lead baseline and the extractive oracle system ext-oracle. Results are computed on the test set.
['Datasets', '% of novel n-grams in gold summary', '% of novel n-grams in gold summary', '% of novel n-grams in gold summary', '% of novel n-grams in gold summary', 'lead', 'lead', 'lead', 'ext-oracle', 'ext-oracle', 'ext-oracle']
We report the percentage of novel n-grams in the target gold summaries that do not appear in their source documents. There are 36% novel unigrams in the XSum reference summaries compared to 17% in CNN, 17% in DailyMail, and 23% in NY Times. This indicates that XSum summaries are more abstractive. The proportion of novel constructions grows for larger n-grams across datasets, however, it is much steeper in XSum whose summaries exhibit approximately 83% novel bigrams, 96% novel trigrams, and 98% novel 4-grams (comparison datasets display around 47–55% new bigrams, 58–72% new trigrams, and 63–80% novel 4-grams). The lead baseline performs extremely well on CNN, DailyMail and NY Times confirming that they are biased towards extractive methods. ext-oracle further shows that improved sentence selection would bring further performance gains to extractive approaches. Abstractive systems trained on these datasets often have a hard time beating the lead, let alone ext-oracle, or display a low degree of novelty in their summaries See et al. Interestingly, lead and ext-oracle perform poorly on XSum underlying the fact that it is less biased towards extractive methods.