What is this Article about? Extreme Summarization with Topic-aware Convolutional Neural Networks
Figure 5: Summaries ranked from least to most informative from the XSum and Newsroom-Abs test sets. We also show their informativeness scores (higher is better).
['Reference Summaries', 'Reference Summaries', 'Score']
We measure the informativeness of each dataset as the average score assigned by crowdworkers across summaries; a summary receives a score of 3 if it is deemed informative, 2 if it is partially informative, and 1 if it is uninformative. Therefore, the informativeness score for a dataset as a whole may vary from 1 to 3, with 1 being least informative and 3 being most informative. We also report the proportion of times a dataset was considered informative, partially informative, and uninformative. XSum reference summaries were mostly considered informative (68%), around a quarter of them (26%) were deemed partially informative, and only 6% were found uninformative. In comparison, less than half (48.67%) of Newsroom-Abs summaries were found informative, the remaining being either partially informative or uninformative. XSum achieved an informativeness score of 2.62 compared to 2.30 for the Newsroom dataset. We carried out pairwise comparisons to assess whether informativeness differences are statistically significant. We found that XSum is significantly more informative than Newsroom (using a one-way ANOVA with posthoc Tukey HSD tests; p<0.01). For both datasets, as the size of n-grams increases, their chance of having been seen in the training summaries decreases rapidly. For XSum, this percentage drops to almost 0% for any n-grams larger than size 10. Interestingly, this is not the case for Newsroom-Abs, where more than 4% of n-grams (sizes 10 to 15) in test summaries have been already seen in training summaries. This result suggests that Newsroom-Abs summaries are somewhat formulaic displaying a certain degree of repetition that goes beyond simple phrases. which is rather generic and would apply to any movie, not just the Exorcist. In fact two crowdworkers labeled this summary as uninformative and one as partially informative (informativeness score is 1.33). XSum summaries exhibit more variation for n-grams of size larger than 5 corraborating our claim that Newsroom-Abs summaries are more repetitive.