BiSET: Bi-directional Selective Encoding with Template for Abstractive Summarization
Table 6: Results of human evaluation.
['Model', 'Info', 'Concise', 'Read']
We then carried out a human evaluation to evaluate the generated summaries from another perspective. Our evaluators include 8 graduate students and 4 senior undergraduates, while the dataset is 100 randomly-selected articles from the test set. Cao et al. and BiSET under three settings, respectively, and 3 randomly-selected summaries for trapping. We asked the evaluators to independently rate each summary on a scale of 1 to 5, with respect to its quality in informativity, conciseness, and readability. While collecting the results, we rejected the samples in which more than half evaluators rate the informativity of the reference summary below 3. We also rejected the samples in which the informativity of a randomly-selected summary is scored higher than 3. Finally, we obtained 43 remaining samples and calculated an average score for each aspect.