Make Lead Bias in Your Favor: A Simple and Effective Method for News Summarization
Table 5: Average and standard deviations of human evaluation scores for summaries on CNN/DailyMail test set. Scores range from 1 to 5 with 5 being perfect. Each summary is judged by 3 human evaluators. PL-FT’s result is statistically significant compared with pointer-generator network with coverage with a p-value less than 10−7.
['[BOLD] Model', 'Average Score', 'Standard deviation']
We conduct human evaluation of the generated summaries from our models and the pointer generator network with coverage. We randomly sample 100 articles from the CNN/DailyMail test set and ask 3 human labelers from Amazon Mechanical Turk to assess the quality of summaries with a score from 1 to 5 (5 means perfect quality. The labelers need to judge whether the summary can express the salient information from the article in a concise form of fluent language. To reduce bias, we randomly shuffle summaries from different sources for each article. the pointer generator network (PTGen+Cov), and PL-FT’s advantage over PTGen+Cov is statistically significant. This shows the effectiveness of both our pretraining and finetuning strategy. To evaluate the inter-annotator agreement, we compute the kappa statistics among the labels and the score is 0.34.