On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
Table 2: Summarization results on the arXiv dataset. Previous work results from [discourse/corr/abs-1804-05685]. The following lines are a simple baseline Lead-10 extractor and the pointer and classifier models. Our transformer LMs (TLM) are conditioned either on the Introduction (I) or along with extracted sentences (E) either from ground-truth (G) or model (M) extracts.
['Model', 'Type', 'ROUGE 1', 'ROUGE 2', 'ROUGE 3', 'ROUGE L']
Our extractive models are able to outperform previous extractive baselines on both the arXiv and Pubmed datasets. Our TLM conditioned on the extractive summary produced by our best extractive model (TLM-I+E (G,M)) outperforms prior abstractive/mixed results on the arXiv, Pubmed and bigPatent datasets, except on ROUGE-L. On Newsroom, we do better than the only other abstractive model (Seq2Seq with attention) by a massive margin and achieve better performance than the pointer generator even on the abstractive and mixed which their model should be better suited to since it has a copy mechanism. The Exconsumm model [ mendes2019jointly] however, which is primarily an extractive model does better on this dataset. We suspect the poor ROUGE-L result is due to the absence of a copy mechanism that makes it hard to get exact large n-gram matches. Further, [graham2015re] finds that ROUGE-L is poorly correlated with human judgements when compared to ROUGE-1,2,3. We show a performance upper bound conditioning the Transformer LM on oracle / ground-truth extracted sentences at both train and test time (TLM-I+E (G,G)). We also experiment with using either the ground-truth extracted sentences (TLM-I+E (G,M)) or the model extracted sentences (TLM-I+E (M,M)) during training and find that latter slightly impairs performance.