Towards Annotating and Creating Sub-Sentence Summary Highlights
Table 2: ROUGE results on CNN/DM test set at both sentence and sub-sentence level. The top two rows test gold-standard sentences and sub-sentences against human abstracts. Additionally we show results of an abstractive See et al. (2017) and an extractive summarizer Arumae and Liu (2019) whose CNN/DM results are macro-averaged. The bottom two sections showcase our models. We report results at sentence and sub-sentence level and report those with and without E\scriptsize d-pos embeddings (+posit.). These results are further broken down to reflect evaluation against human abstracts and our own gold standard segments.
['Model', '[BOLD] ROUGE-1 P', '[BOLD] ROUGE-1 R', '[BOLD] ROUGE-1 F1', '[BOLD] ROUGE-2 P', '[BOLD] ROUGE-2 R', '[BOLD] ROUGE-2 F1', '[BOLD] ROUGE-L P', '[BOLD] ROUGE-L R', '[BOLD] ROUGE-L F1']
These are the oracle results for sentences and segments, respectively. Despite that abstracts can contain unseen words, we observe that 70% of the abstract words are covered by gold-standard sentences, and 51% of abstract words are included in sub-sentence units, suggesting the effectiveness of our annotation method on capturing summary-worthy content. We obtain system-predicted whole sentences (Sent) and sub-sentence segments (Segm); then evaluate them against both human abstracts (Abstract) and gold-standard highlights (Sub-sent). We test the efficacy of document positional embeddings (Eq.