Synthesizer: Rethinking Self-Attention in Transformer Models
Table 4: Experimental results (dev scores) on multi-task language understanding (GLUE benchmark) for small model and en-mix mixture. Note: This task has been co-trained with SuperGLUE.
['Model', 'SGlue', 'BoolQ', 'CB', 'CoPA', 'MultiRC', 'ReCoRD', 'RTE', 'WiC', 'WSC']
We note that the (R) and (D) variants of Synthesizer do not achieve reasonable performance. This can be largely attributed to the fact that the encoder self-attention in the T5 setting also functions as a cross-sentence attention. For example, in the entailment or reading comprehension tasks, the premise and hypothesis are concatenated together and self-attention effectively acts as cross-sentence attention. Optimistically, we observe that Syn (R+V) outperforms the T5 model by a substantial margin (+1.9 points on SuperGLUE and +0.6 points on GLUE).