Contrastive Attention Mechanism for Abstractive Sentence Summarization
Table 3: Results of explorations on the opponent attention derivation. The upper part presents the influence of masking more attention weights for deriving the opponent attention. The middle part presents the results of selecting different head for the opponent attention derivation. The bottom row presents the result of Transformer.
['System', 'Gigaword R-1', 'Gigaword R-2', 'Gigaword R-L', 'DUC2004 R-1', 'DUC2004 R-2', 'DUC2004 R-L']
It shows that masking maximum weight performs better, indicating that masking the most salient weight leaves more irrelevant or less relevant words to compute the opponent probability Po, which is more reliable than that computed from less remaining words after masking more weights. “non-synchronous head” performs worst, and even worse than the Transformer baseline on Gigaword. This indicates that it is better to compose the opponent attention from irrelevant parts that can be easily located in the synchronous head. “averaged head” performs slightly worse than “synchronous head”, and is also slower due to the involved all heads.