Bridging the Gap for Tokenizer-Free Language Models
Table 2: Comparing recent language model results on lm1b.
['[EMPTY]', 'Segmentation', 'Context Length', '# of params', 'Perplexity', 'Bits/Byte']
We observe that tokenizer-free LM performance improves significantly (40.6 to 23.0) when the model capacity is increased from 0.2B to 0.8B parameters. With sufficient capacity our byte-level LM is competitive with word based models (ranging from 21.8 to 28.0). Note, our model is able to achieve comparable performance without any explicit signal of word boundaries.