Neural Network-Based Abstract Generation for Opinions and Arguments
Table 3: Automatic evaluation results by BLEU, METEOR, and ROUGE SU-4 scores (multiplied by 100) for abstract generation systems. The average lengths for human written summaries are 11.5 and 24.6 for RottenTomatoes and Idebate. The best performing system for each column is highlighted in boldface, where our system with pre-trained word embeddings and additional features achieves the best BLEU scores on both datasets. Our systems that are statistically significantly better than the comparisons are highlighted with ∗ (two-tailed Wilcoxon signed rank test, p<0.05). Our system also has the best METEOR and ROUGE scores (in italics) on RottenTomatoes dataset among learning-based systems.
['[EMPTY]', '[ITALIC] RottenTomatoes  [BOLD] Length', '[ITALIC] RottenTomatoes  [BOLD] BLEU', '[ITALIC] RottenTomatoes  [BOLD] METEOR', '[ITALIC] RottenTomatoes  [BOLD] ROUGE', '[ITALIC] Idebate  [BOLD] Length', '[ITALIC] Idebate  [BOLD] BLEU', '[ITALIC] Idebate  [BOLD] METEOR', '[ITALIC] Idebate  [BOLD] ROUGE']
Our system with pre-trained word embeddings and additional features achieves the best BLEU scores on both datasets (in boldface) with statistical significance (two-tailed Wilcoxon signed rank test, p<0.05). Notice that our system summaries are conciser (i.e. shorter on average), which lead to higher scores on precision based-metrics, e.g. BLEU, and lower scores on recall-based metrics, e.g. METEOR and ROUGE. On RottenTomatoes dataset, where summaries generated by different systems are similar in length, our system still outperforms other methods in METEOR and ROUGE in addition to their significantly better BLEU scores. This is not true on Idebate, since the length of summaries by extract-based systems is significantly longer. But the BLEU scores of our system are considerably higher. Among our four systems, models with pre-trained word embeddings in general achieve better scores. Though additional features do not always improve the performance, we find that they help our systems converge faster.