Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Table 3: Comparison with state-of-the-art results on text8.
['[BOLD] Model', '[BOLD] #Param', '[BOLD] bpc']
Similar to but different from enwik8, text8 contains 100M processed Wikipedia characters created by lowering case the text and removing any character other than the 26 letters a through z, and space. Due to the similarity, we simply adapt the best model and the same hyper-parameters on enwik8 to text8 without further tuning. Again, Transformer-XL achieves the new SoTA result with a clear margin.