Learning to Create Sentence Semantic Relation Graphs for Multi-Document Summarization
Table 1: Comparison of various models using ROUGE-1/ROUGE-2 on DUC 2004 with 665 bytes summaries.
['Model', 'ROUGE-1', 'ROUGE-2']
We compare SemSentSum with three types of model relying on either 1) sentence or document embeddings 2) various hand-crafted features or 3) additional data. For the first category, we significantly outperform MMR Bennani-Smires et al. PV-DBOW+BS Mani et al. Although their methods are based on embeddings to represent the meaning, it shows that using only various distance metrics or encoder-decoder architecture on these is not efficient for the task of multi-document summarization (as also shown in the Ablation Study). We hypothesize that SemSentSum performs better by leveraging pre-trained sentence embeddings and hence lowering the effects of data scarcity. Systems based on hand-crafted features include a widely-used learning-based summarization method, built on support vector regression SVR Li et al. ; Peer 65 which is the best peer systems participating in DUC evaluations; and the recursive neural network R2N2 of Cao et al. that learns automatically combinations of hand-crafted features. As can be seen, among these models completely dependent on hand-crafted features, SemSentSum achieves highest performance on both ROUGE scores. This denotes that using different linguistic and word-based features might not be enough to capture the semantic structures, in addition to being cumbersome to craft. The last type of model is shown in TCSum Cao et al. In terms of ROUGE-1, SemSentSum significantly outperforms TCSum and performs similarly on ROUGE-2 score. This demonstrates that collecting more manually annotated data and training two models is unnecessary, in addition to being difficult to use in other domains, whereas SemSentSum is fully data driven, domain-independent and usable in realistic scenarios.