SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
Table 3: Performance comparison of abstractive and extractive models on the entire CNN Daily Mail test set using full-length F1 variants of Rouge. SummaRuNNer is able to significantly outperform the abstractive state-of-the-art as well as the Lead-3 baseline (on Rouge-1 and Rouge-2).
['[EMPTY]', 'Rouge-1', 'Rouge-2', 'Rouge-L']
We also report the performance of SummaRuNNer on the joint CNN/Daily Mail corpus. In order to do a fair comparison with their work, we use the same metric as them. The superior performance of our model is not entirely surprising since abstractive summarization is a much harder problem, but the table serves to quantify the current performance gap between extractive and abstractive approaches to summarization. The results also demonstrate the difficulty of using the F1 metric for extractive summarization since SummaRuNNer, with its top three sentences with highest prediction probability as the summary, errs on the side of high recall at the expense of precision. Dynamically adjusting the summary length based on predicted probability distribution may help balance precision and recall and may further boost F1 performance, but we have not experimented with it in this work.