Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Table 2: Comparison with state-of-the-art results on enwik8.
['[BOLD] Model', '[BOLD] #Param', '[BOLD] bpc']
The dataset enwik8 contains 100M bytes of unprocessed Wikipedia text. Under the model size constraint, the 12-layer Transformer-XL achieves a new SoTA result, outperforming the 12-layer vanilla Transformer from Al-Rfou et al. Notably, our 12-layer architecture achieves the same result as the 64-layer network from Al-Rfou et al. In order to see whether better performances can be obtained by increasing the model size, we train 18-layer and 24-layer Transformer-XLs with increased model sizes. With the attention length 784 during training and 3,800 during evaluation, we obtained a new SoTA result and our method is the first to break through 1.0 on widely-studied character-level benchmarks. Different from Al-Rfou et al.