Neural Network-Based Abstract Generation for Opinions and Arguments
Table 4: Human evaluation results for abstract generation systems. Inter-rater agreement for overall ranking is 0.71 by Krippendorff’s α. Informativeness (Info), grammaticality (Gram), and Compactness (Comp) are rated on a 1 to 5 scale, with 5 as the best. Our system achieves the best informativeness and grammaticality scores among the three learning-based systems. Our summaries are ranked as the best in 18% of the evaluations, and are also ranked higher than compared systems on average.
['[EMPTY]', '[BOLD] Info', '[BOLD] Gram', '[BOLD] Comp', '[BOLD] Avg Rank', '[BOLD] Best%']
As it can be seen, our system outperforms the abstract-based system Opinosis in all aspects, and also achieves better informativeness and grammaticality scores than LexRank, which extracts sentences in their original form. Our system summaries are ranked as the best in 18% of the evaluations, and has an average ranking of 2.3, which is higher than both Opinosis and LexRank on average. An inter-rater agreement of Krippendorff’s α of 0.71 is achieved for overall ranking. This implies that our attention-based abstract generation model can produce summaries of better quality than existing summarization systems. We also find that our system summaries are constructed in a style closer to human abstracts than others.