Concept Pointer Network for Abstractive Summarization
Table 4: Human evaluation: scoring of three models in terms of abstraction and overall quality by human evaluators (the higher the better). The score range could be 0-20. ⋆ indicates the improvements from the baselines to the concept pointer are statistically significant.
['Method', 'abstraction', 'overall quality']
To explore the correctness of our model using human judgment, we conducted a manual evaluation with 20 post-graduate volunteers. We primarily used the following criteria to assess the generated summaries: abstraction, i.e., Are the abstract concepts contained in the summary appropriate?; and overall quality, i.e., Is the summary readable, informative, relevant, etc.? To conduct the evaluation, we randomly selected 20 examples from the DUC 2004 testing set and asked the volunteers to subjectively assess the summaries. Each example consisted of an article and three summaries, i.e., a summary by the seq2seq model, the pointer generator model, and our proposed concept pointer model. The volunteers chose the best summaries for each of the articles according to the above criteria (can be multiple choices). Obviously, the summaries were randomly shuffled, and the model used to produce each was unknown to prevent bias. The scores for each model were ranked by how many times the volunteers chose a summary w.r.t each criteria, averaged by the number of participants.