Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Table 5: Proportion of novel n-grams in summaries generated by various models on the XSum test set.
['Models', '% of novel n-grams in generated summaries', '% of novel n-grams in generated summaries', '% of novel n-grams in generated summaries', '% of novel n-grams in generated summaries']
We further assessed the extent to which various models are able to perform rewriting by generating genuinely abstractive summaries. As can be seen, the convolutional models exhibit the highest proportion of novel n-grams. We should also point out that the summaries being evaluated have on average comparable lengths; the summaries generated by PtGen contain 22.57 words, those generated by ConvS2S and T-ConvS2S have 20.07 and 20.22 words, respectively, while gold summaries are the longest with 23.26 words. Interestingly, PtGen trained on XSum only copies 4% of 4-grams in the source document, 10% of trigrams, 27% of bigrams, and 73% of unigrams. This is in sharp contrast to PtGen trained on CNN/DailyMail exhibiting mostly extractive patterns; it copies more than 85% of 4-grams in the source document, 90% of trigrams, 95% of bigrams, and 99% of unigrams See et al. This result further strengthens our hypothesis that XSum is a good testbed for abstractive methods.