Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Table 6: Ablation study on WikiText-103. For the first two blocks, we use a slightly smaller model (128M parameters). † indicates that the corresponding row is reduced to the same setting as the Transformer network in Al-Rfou et al. (2018), except that two auxiliary losses are not implemented in our experiments. “PPL init” refers to using the same length as training. “PPL best” indicates the perplexity obtained by using the optimal length. “Attn Len” is the shortest possible attention length during evaluation to achieve the corresponding result (PPL best). Increasing the attention length during evaluation improves performance only when our positional encoding is used. The “Transformer-XL (151M)” setting uses a standard parameter budget as previous work Merity et al. (2018), where we observe a similar effect when increasing the attention length during evaluation.
['[BOLD] Remark', '[BOLD] Recurrence', '[BOLD] Encoding', '[BOLD] Loss', '[BOLD] PPL init', '[BOLD] PPL best', '[BOLD] Attn Len']
The first study is performed on WikiText-103, which requires modeling long-term dependency. Among the compared encoding schemes, Shaw et al. “Full” and “half” losses refer to applying a cross entropy loss to all or the recent half positions in the segment. We found that absolute encodings only work well with half losses because half losses exclude positions with very short attention lengths during training for better generalization. Although the backpropagation length during training is only 128, with the two techniques the attention length can be increased to 640 at test time. In the standard setting with 151M parameters, the perplexity decreases as the attention length increases.