Synthesizer: Rethinking Self-Attention in Transformer Models
Table 3: Experimental results on Abstractive Summarization (CNN/Dailymail) and Dialogue Generation (PersonaChat).
['Model', 'Summarization Rouge-1', 'Summarization Rouge-2', 'Summarization Rouge-L', 'Dialogue Bleu-1/4', 'Dialogue Rouge-L', 'Dialogue Meteor', 'Dialogue CIDr', 'Dialogue Emb']
For summarization, we find that the (R) and (D) variants do not outperform Transformers. The performance of the (D) model is ≈2 Rouge-L points below Transformers. Hence, we postulate that the local sample-wise pairwise interactions are important for the summarization task. On the other hand, the utility of synthesized attention can also be observed, i.e., the (R+V) and (R+D) models both outperform Transformers.