A Mixture of h-1 Heads is Better than h Heads
Table 4: Language modeling performance on WikiText-103 test set (lower is better). ⋆Trains/evaluates with 3,072/2,048 context sizes and therefore not directly comparable to other models which use 512/480 sized ones. See Table 2 caption for the indications of other superscripts. Bold font indicates the best performance using smaller context sizes. The first two rows are due to Table 5 of baevski2018adaptive.
['[BOLD] Model', '[BOLD] Perplexity', '[BOLD] # Params.']
S4SS3SSS0Px4 Results. When trained under the same setting, MAE outperforms baevski2018adaptive by more than 0.3 perplexity. Interestingly, despite the much smaller context at both training and test time, MAE matches the best setting by baevski2018adaptive. uni-Mae-7 and noBCD underperform the baseline (higher perplexity).