Improving Transformer Models by Reordering their Sublayers
Table 3: Performance on the WikiText-103 test set. We compare the best sandwich transformer to the unmodified, interleaved transformer baseline baevski2018adaptive trained over 5 random seeds and to other previously reported results.
['[BOLD] Model', '[BOLD] Test']
This improvement comes at no extra cost in parameters, data, memory, or computation; we did not even change any of the original hyperparameters, including the number of training epochs. To check whether this advantage is consistent, we train 4 more sandwich166 models with different random seeds (5 in total) and evaluate them on the development set, to avoid evaluating our model more than once on the test set. This is the only experiment in which we modify our model’s random seed.