HipoRank: Incorporating Hierarchical and Positional Information into Graph-based Unsupervised Long Document Extractive Summarization
Table 4: Human evaluation results based on 20 sampled reference summaries with 281 system summary sentences. Each reference summary-sentence pair is annotated by two annotators with an average annotator agreement of 73.24%. The results are averaged across 127 sentences from HipoRank and 154 sentences from state-of-the-art unsupervised extractive summarization system PACSUM (Zheng and Lapata, 2019).
['% of YES', 'content-coverage', 'importance']
HipoRank is shown to be significantly better than PACSUM in terms of both content coverage (p=0.004) and importance (p=0.014), with high inter-annotator agreements (avg. 73.24%). These results indicate that the use of discourse structure to guide the graph construction with our proposed boundary function and hierarchy information improves summarization quality.