A Mixture of h-1 Heads is Better than h Heads
Table 3: IWSLT14 GE-DE test set performance. See Table 2 caption for indications of the superscripts.
['[BOLD] Model', '[BOLD] BLEU', '[BOLD] # Params.']
The Base model here is similar to the base-sized transformer in the WMT14 experiment, but with a smaller hidden dimension. Mae-7 outperforms Base by 0.9 BLEU. Interestingly, uni-Mae-7 improves over Base by 0.3 BLEU, possibly because the regularization effect of random expert selection training helps more on this smaller dataset.