Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization
Table 5: Linguistic quality and rankings of system summaries. (DUC-04)
['[BOLD] System', '[BOLD] Linguistic Quality  [BOLD] Fluency', '[BOLD] Linguistic Quality  [BOLD] Inform.', '[BOLD] Linguistic Quality  [BOLD] NonRed.', '[BOLD] Rankings (%)  [BOLD] 1st', '[BOLD] Rankings (%)  [BOLD] 2nd', '[BOLD] Rankings (%)  [BOLD] 3rd', '[BOLD] Rankings (%)  [BOLD] 4th']
Linguistic quality. To assess the linguistic quality of various system summaries, we employ Amazon Mechanical Turk human evaluators to judge the summary quality, including PG-MMR, LexRank, PG-Original, and Extract+Rewrite. A turker is asked to rate each system summary on a scale of 1 (worst) to 5 (best) based on three evaluation criteria: informativeness (to what extent is the meaning expressed in the ground-truth text preserved in the summary?) , fluency (is the summary grammatical and well-formed?), and non-redundancy (does the summary successfully avoid repeating information?). Human summaries are used as the ground-truth. The turkers are also asked to provide an overall ranking for the four system summaries. We observe that the LexRank summaries are highest-rated on fluency. This is because LexRank is an extractive approach, where summary sentences are directly taken from the input. PG-MMR is rated as the best on both informativeness and non-redundancy. Regarding overall system rankings, PG-MMR summaries are frequently ranked as the 1st- and 2nd-best summaries, outperforming the others.