A Mixture of h-1 Heads is Better than h Heads
Table 2: WMT14 EN-DE translation test performance on newstest2014. † randomly select an expert to update for each training instance, and ‡ learns a gating function to weight the experts. Transformer performance in the first two rows are due to vaswani2017attention.
['[BOLD] Model', '[BOLD] BLEU', '[BOLD] # Params.']
The base and large sized transformer models are due to vaswani2017attention. To control for compounding factors, we additionally compare to our implementation of the base sized model (Base). It achieves slightly better performance than vaswani2017attention, with a 0.3 BLEU edge. Mae-7 improves over the base transformer by 0.8 BLEU, obtaining similar performance to the large-size transformer of vaswani2017attention using less than a third as many parameters. Since we do not see similar improvement by uni-Mae-7, we attribute this gain to input-dependent expert weighting. Having a smaller number of heads for each expert, Mae-6 slightly underperforms Mae-7, and so does uni-Mae-6 in comparison to uni- Mae-7. Finally, noBCD gets worse performance than the transformer baseline, demonstrating the importance of the block coordinate decent training.