Synthesizer: Rethinking Self-Attention in Transformer Models
Table 2: Experimental Results on WMT’14 English-German, WMT’14 English-French Machine Translation tasks and Language Modeling One Billion (LM1B).
['Model', 'NMT (BLEU) # Params', 'NMT (BLEU) EnDe', 'NMT (BLEU) EnFr', 'LM (PPL) # Params', 'LM (PPL) LM1B']
First, we observe that our Random Synthesizer baseline achieves 27.27 on EnDe and 41.12 on EnFr. The non-trainable (i.e., fixed) variant performs substantially worse, but still yields surprisingly strong ≈24 BLEU with fixed random attention weights. Most other Synthesizer variants achieve competitive performance, although with slight performance degradation compared to Transformers. An interesting finding is that the Mixture model of Random and Dense synthesizer outperforms vanilla Transformers on EnDe. When mixing the standard dot product attention, performance further increases by +0.8 BLEU points on EnDe. We find that the Random Synthesizers perform within 1−2 perplexity points away from the vanilla Transformer model. The best performing model is the Synthesizer (Dense + Vanilla), which achieves the best performance on this setting.