Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Table 4: Comparison with state-of-the-art results on One Billion Word. ⋄ indicates contemporary work.
['[BOLD] Model', '[BOLD] #Param', '[BOLD] PPL']
One Billion Word does not preserve any long-term dependency because sentences have been shuffled. Consequently, this dataset mainly tests the ability of modeling only short-term dependency. Although Transformer-XL is mainly designed to better capture longer-term dependency, it dramatically improves the single-model SoTA from 23.7 to 21.8. Specifically, Transformer-XL significantly outperforms a contemporary method using vanilla Transformers Baevski and Auli suggesting the advantage of Transformer-XL is generalizable to modeling short sequences.