Hierarchical Transformers for Multi-Document Summarization
Table 2: Test set results on the WikiSum dataset using ROUGE F1.
['Model', 'ROUGE-1', 'ROUGE-2', 'ROUGE-L']
The first block in the table includes extractive systems (Lead, LexRank), the second block includes several variants of Flat Transformer-based models (FT, T-DMCA), while the rest of the table presents the results of our Hierarchical Transformer (HT). As can be seen, abstractive models generally outperform extractive ones. The Flat Transformer, achieves best results when the input length is set to 800 tokens, while longer input (i.e., 1,200 tokens) actually hurts performance. The Hierarchical Transformer with 1,600 input tokens, outperforms FT, and even T-DMCA when the latter is presented with 3,000 tokens. Adding an external graph also seems to help the summarization process. The similarity graph does not have an obvious influence on the results, while the discourse graph boosts ROUGE-L by 0.16. We also found that the performance of the Hierarchical Transformer further improves when the model is presented with longer input at test time. This suggests that the model can potentially generate better summaries without increasing training time.