Attend to the beginning: A study on using bidirectional attention for extractive summarization
Table 3: ROUGE results over MSW dataset.
['[BOLD] Model', '[BOLD] R-1', '[BOLD] R-2', '[BOLD] R-L', '[EMPTY]']
The motivation behind conducting experiments on the MSW dataset is to validate our last hypothesis (H4). It shows that attending to the beginning of a document helps selecting more salient sentences, not just for discussion threads, but even for generic textual documents. Similar to the results on the discussions dataset, we can see that attending to the beginning through a bidirectional attention mechanism boosts ROUGE scores (rows 1, and 2). Additionally, we can see that combining bidirectional attention with BERT embeddings further improves the performance (rows 1, and 4). Surprisingly, it was only able to outperform the lead3 baseline. Through analyzing different criteria of the generated output for SummaRuNNer and SiATL, over the two used datasets. We observed that SiATL tends to generate longer summaries compared to SummaRuNNer, and this most likely due to its non-auto-regressive nature. SummaRuNNer, on the other hand, tends to generate shorter summaries. It shows that for the forum discussion dataset, the expected summary length is ∼ 14 sentences. For the same dataset, the SiATL model generates summaries of length ∼ 16 sentences, while SummaRuNNer generates summaries of length ∼ 8 sentences. This can justify the superior performance of SiATL compared to SummaRuNNer on the forum discussion dataset. On the other hand, we can notice that the expected summary length for the MSW dataset is ∼ 8 sentences. For the same dataset, SummaRuNNer consistently generates shorter summaries compared to SiATL of lengths ∼ 6.5 compared to 22 respectively. It is clear that the huge difference in the length of the summary between the human and SiATL generated is the reason SiATL underperforms on the MSW dataset. A potential solution for the SiATL model would be by adding a final post-processing step (e.g. clustering, redundancy reduction, etc..). The rule of the post-processing step would be slightly filtering the generated summary, and help to pick a number of sentences close to the average number humans select.