What is this Article about? Extreme Summarization with Topic-aware Convolutional Neural Networks
Table 11: XSum and Newsroom-Abs summaries and their informativeness. The middle column proportionately shows the number times a summary was judged “Informative”, “Partially Informative”, or “Uninformative.” The last column shows the informativeness score for each dataset (higher is better).
['Dataset', 'Informative', 'Part. Informative', 'Uninformative', 'Score']
We measure the informativeness of each dataset as the average score assigned by crowdworkers across summaries; a summary receives a score of 3 if it is deemed informative, 2 if it is partially informative, and 1 if it is uninformative. Therefore, the informativeness score for a dataset as a whole may vary from 1 to 3, with 1 being least informative and 3 being most informative. We also report the proportion of times a dataset was considered informative, partially informative, and uninformative. XSum reference summaries were mostly considered informative (68%), around a quarter of them (26%) were deemed partially informative, and only 6% were found uninformative. In comparison, less than half (48.67%) of Newsroom-Abs summaries were found informative, the remaining being either partially informative or uninformative. XSum achieved an informativeness score of 2.62 compared to 2.30 for the Newsroom dataset. We carried out pairwise comparisons to assess whether informativeness differences are statistically significant. We found that XSum is significantly more informative than Newsroom (using a one-way ANOVA with posthoc Tukey HSD tests; p<0.01).