MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Table 7: Comparison of different tokenization strategies for different languages on MLDoc.
['[EMPTY]', 'de', 'es', 'fr', 'it', 'ru']
Tokenization Subword tokenization has been found useful for language modeling with morphologically rich languages Czapla et al. We train models with the best performing vocabulary sizes for subword (15k) and regular word-based tokenization (60k) with the Moses tokenizer Koehn et al. Subword tokenization outperforms word-based tokenization on most languages, while being faster to train due to the smaller vocabulary size.