Depth-Adaptive Transformer
Table 1: Aligned vs. mixed training on IWSLT De-En. We report valid BLEU for a uniformly sampled exit n∼U([1..6]) at each token, a fixed exit n∈[1..6] for all tokens, as well as the average BLEU over the fixed exits. As baseline we show six standard Transformer models with 1-6 blocks.
['[EMPTY]', 'Uniform', '[ITALIC] n=1', '[ITALIC] n=2', '[ITALIC] n=3', '[ITALIC] n=4', '[ITALIC] n=5', '[ITALIC] n=6', 'Average']
The latter is surprising since aligned training never exposes the self-attention mechanism to hidden states from other blocks. Aligned training also performs very competitively to the individual baseline models.