\DATASET: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Table 5: Average performance of systems as scored by human evaluators. Each summary was scored by three different evaluators. Dimensions, from left to right: informativeness, relevance, fluency, and coherence, and a mean of the four dimensions for each system.
['[EMPTY]', 'Semantic INF', 'Semantic REL', 'Syntactic FLU', 'Syntactic COH', 'Avg.']
No summarization system exceeded the scores given to the Lede-3 baseline. However, the extractive oracle designed to maximize n-gram based evaluation performed worse than the majority of systems under human evaluation. While the fully abstractive Abs-N model performed very poorly under automatic evaluation, it fared slightly better when scored by humans. TextRank received the highest overall score. TextRank generates full sentences extracted from the article, and raters preferred TextRank primarily for its fluency and coherence. The pointer-generator models do not have this advantage, and raters did not find the pointer-generator models to be as syntactically sound as TextRank. However, raters preferred the informativeness and relevance of the Pointer-S and Pointer-N models, though not the Pointer-C model, over TextRank.