Conditional Self-Attention for Query-based Summarization
Table 1: Query-based summarization on Debatepedia (abstractive) and HotpotQA (extractive). Two CSA models are evaluated: (Mul) and (Add) refer to multiplicative and additive cross-attention used in CSA.
['Model', 'Debatepedia\xa0(Nema et al.,  2017 )', 'Debatepedia\xa0(Nema et al.,  2017 )', 'Debatepedia\xa0(Nema et al.,  2017 )', 'HotpotQA\xa0(Yang et al.,  2018 )', 'HotpotQA\xa0(Yang et al.,  2018 )', 'HotpotQA\xa0(Yang et al.,  2018 )']
Note our models have much higher Rouge-2 scores than baselines, which suggests the summarization generated by CSA is more coherent. The learned attention scores emphasize not only lexical units such as ”coal-electricity” but also conjunctive adverb such as ”therefore.” More example summaries can be found in Appendix. Different from Debatepedia results, CSA module with multiplicative cross-attention performs better than the additive one. This is because in each dataset we adopt the same hyper-parameters for both variants – in Debatepedia the hyper-parameters favor multiplicative cross-attention, while in HotpotQA they prefer the additive one. The experimental results show that both variants can have significantly better performance than the baselines. It also suggests that we cannot simply adopt the hyper-parameters from another tasks without fine-tuning – although the model can be more compelling than the baselines if we do so.