Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Table 7: Ablation study on One Billion Word, a dataset without long-term dependency.
['[BOLD] Method', '[BOLD] PPL']
The second study targets at isolating the effects of resolving the context fragmentation problem from the benefit of capturing longer context length. In order to achieve this goal, we deliberately choose a dataset that does not require long-term dependency, so that any improvement from establishing the recurrence can be attributed to solving the context fragmentation. Specifically, we perform this controlled experiment on the One Billion Word dataset, which can only benefit from removing the context fragmentation. We train a 20-layer Transformer-XL with ∼0.3B parameters for 400K steps. Moreover, our relative positional encodings is also superior to Shaw et al.