Toward Extractive Summarization of Online Forum Discussions via Hierarchical Attention Networks
Table 1: Results of thread summarization. ‘HAN’ models are our proposed approaches adapted from the hierarchical attention networks [Yang et al.2016]. The models can be pretrained using unlabeled threads from TripAdvisor (‘T’) and Ubuntuforum (‘U’). r indicates a redundancy removal step is applied. We report the variance of F-scores across all threads (‘±’). A redundancy removal step improves recall scores (shown in gray) of the HAN models and boosts performance.
['[BOLD] System', '[BOLD] ROUGE-1  [BOLD] R (%)', '[BOLD] ROUGE-1  [BOLD] P (%)', '[BOLD] ROUGE-1  [BOLD] F (%)', '[BOLD] ROUGE-2  [BOLD] R (%)', '[BOLD] ROUGE-2  [BOLD] P (%)', '[BOLD] ROUGE-2  [BOLD] F (%)', '[BOLD] Sentence-Level  [BOLD] R (%)', '[BOLD] Sentence-Level  [BOLD] P (%)', '[BOLD] Sentence-Level  [BOLD] F (%)']
The experimental results of all models are shown in Table 1. [CONTINUE] First, HAN models appear to be more appealing than SVM and LogReg because there is less variation in program implementation, hence less effort is required to reproduce the results. HAN models outperform both LogReg and SVM using the current set of features. They yield higher precision scores than traditional models. [CONTINUE] With respect to ROUGE scores, the HAN models outperform all supervised and unsupervised baselines except MEAD. MEAD has been shown to perform well in previous studies (Luo et al. 2016) and it appears to handle redundancy removal exceptionally well. The HAN models outperform MEAD in terms of sentence prediction. [CONTINUE] Pretraining the HAN models, although intuitively promising, yields only comparable results with those without. We suspect that there are not enough data to pretrain the models and that the thread classification task used to pretrain the HAN models may not be sophisticated enough to learn effective thread vectors. [CONTINUE] We observe that the redundancy removal step is crucial for the HAN models to achieve outstanding results. It helps improve the recall scores of both ROUGE and sentence prediction. When redundancy removal was applied to LogReg, it produces only marginal improvement. This suggests that future work may need to consider principled ways of redundancy removal.