MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Table 2: Comparison of zero-shot and supervised methods on MLDoc.
['[EMPTY]', 'de', 'es', 'fr', 'it', 'ja', 'ru', 'zh']
In the zero-shot setting, MultiBERT underperforms the comparison methods as the shared embedding space between many languages is overly restrictive. Our monolingual LMs outperform their cross-lingual teacher LASER in almost every setting. When fine-tuned with only 100 target language examples, they are able to outperform all zero-shot approaches except MultiFiT on de and fr. This calls into question the need for zero-shot approaches, as fine-tuning with even a small number of target examples is able to yield superior performance. When fine-tuning with 1,000 target examples, MultiFiT—even without pretraining—outperforms all comparison methods, including monolingual BERT.