Improving Transformer Models by Reordering their Sublayers
Table 5: Performance on character-level language modeling, evaluated on the enwik8 and text8 test sets. The baseline model Sukhbaatar2019 is trained over 5 random seeds. The sandwich coefficient is tuned on each benchmark’s validation set, and we run our model on the test only once.
['[BOLD] Model', '[BOLD] text8 (BPC)', '[BOLD] enwik8 (BPC)']
We tune the sandwich coefficient on the development set for k∈{1,…,8} (the baseline model has 24 transformer layers). We do not modify any hyperparameters, including the number of training epochs. On text8, the sandwich transformer performs within the baseline’s random seed variance. On enwik8, the sandwich transformer gains an improvement of about 0.007 bits-per-character, matching the state of the art results obtained by the Transformer-XL-based Compressive Transformer of compressive.