A Mixture of h-1 Heads is Better than h Heads
Table 5: Performance decrease for different models on WMT14 development set when only one expert is used for each multi-head attention layer (5.1).
['[BOLD] Model', '[BOLD] BLEU', '[BOLD] Diff.']
We then continue and explore whether MAE performs reasonably well when using only the most “specialized” experts. For each development instance, we select those experts maximizing the gating weights and ignore the rest, instead of linearly combining them as in Eq. In comparison, noBCD has a larger performance decrease of 0.7 BLEU. noBCD ’s performance drop is similar to that of uni-Mae-7, for which we randomly select an expert at each layer and average the performance over 5 runs. These results support the proposition that MAE specializes better when trained with BCD.