Controlling the Amount of Verbatim Copying in Abstractive Summarization
Table 6: Human assessment of informativeness, grammaticality, truthfulness, and best-worst scaling.
['[BOLD] System', '[BOLD] Inform.', '[BOLD] Gramm.', '[BOLD] Truthful.', '[BOLD] Bst-Wst']
To thoroughly analyze the quality of summaries, we ask human annotators to assess system outputs along three dimensions, including informativeness (Has the summary covered important content of the source text?) , grammaticality (Is the summary sentence grammatically correct?), and truthfulness (Has the summary successfully preserved the meaning of the original text?). Both system and human summaries are scored according to these criteria using a Likert scale from 1 (worst) to 5 (best). We compare variants of our method generating (a) pure extracts (case e.) and (b) best abstracts (case g.), baselines of (c ) PG networks, (d) R3Sum, (e) BiSET, and (f) human abstracts. The final rating of the system is computed as the percentage of times it was selected as the best minus that of the worst. We sample 200 instances from the Gigaword test set for evaluation. Each instance was assessed by five human evaluators from Amazon mechnical turk where low-quality annotations are manually removed. We observe that human summaries (article titles) are imperfect. In contrast, both of our systems perform slightly but consistently better than other baselines.