MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Table 6: Bootstrapping results on MLDoc with and without pretraining, trained on 1k/10k LASER labels.
['[EMPTY]', 'de', 'es', 'fr', 'it', 'ja', 'ru', 'zh']
S6SS0SSS0Px3 Robustness to noise We suspect that MultiFiT is able to outperform its teacher as the information from pretraining makes it robust to label noise. To test this hypothesis, we train MultiFiT and a randomly initialized model with the same architecture on 1k and 10k examples of the Spanish MLDoc. The pretrained MultiFiT is able to partially ignore the noise, up to 65% of noisy training examples. Without pretraining, the model does not exceed the theoretical baseline (the percentage of correct examples). Pretraining enables MultiFiT to achieve much better performance compared to a randomly initialised model. Both results together suggest a) that pretraining increases robustness to noise and b) that information from monolingual and cross-lingual language models is complementary.