Outline Generation: Understanding the Inherent Content Structure of Documents
Table 3. Model analysis of our HiStGen model under the automatic evaluation. Two-tailed t-tests demonstrate the improvements of HiStGen to the variants are statistically significant (‡ indicates p-value<0.01).
['Model', 'celebrity EM [ITALIC] outline', 'celebrity EM [ITALIC] sec', 'celebrity Rouge [ITALIC] head', 'cities EM [ITALIC] outline', 'cities EM [ITALIC] sec', 'cities Rouge [ITALIC] head', 'music EM [ITALIC] outline', 'music EM [ITALIC] sec', 'music Rouge [ITALIC] head', 'mixture EM [ITALIC] outline', 'mixture EM [ITALIC] sec', 'mixture Rouge [ITALIC] head']
We conduct ablation analysis to investigate the effect of proposed mechanisms in our HiStGen model. (1) By removing the Markov paragraph dependence mechanism, the performance of HiStGen−P in terms of EMsec has a significant drop as compared with HiStGen. The results indicate that modeling the dependency between adjacent paragraphs does help decide the section boundaries. (2) HiStGen−S performs worse than HiStGen−H and HiStGen−R in terms of Rougehead, showing that the coherence between a section and its heading (captured by the section-aware attention mechanism) has much bigger impact than that between context headings (captured by the Markov heading dependency mechanism and review mechanism) for heading generation. (3) HiStGenPSHR gives the worst performance, indicating that traditional seq2seq model without considering three-level coherence is not suitable for the OG task. (4) By including all the mechanisms, HiStGen achieves the best performance in terms of all the evaluation metrics.