MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
Table 3: Comparison of zero-shot, translation-based and supervised methods (with 2k training examples) on all domains of CLS. MT-BOW and CL-SCL results are from Zhou et al. (2016).
['[EMPTY]', '[EMPTY]', 'de Books', 'de DVD', 'de Music', 'fr Books', 'fr DVD', 'fr Music', 'ja Books', 'ja DVD', 'ja Music']
Cls MultiFiT is able to outperform its zero-shot teacher LASER across all domains. Importantly, the bootstrapped monolingual model also outperforms more sophisticated models that are trained on translations across almost all domains. In the supervised setting, MultiFiT similarly outperforms multilingual BERT. For both datasets, our methods that have been pretrained on 100 million tokens outperform both multilingual BERT and LASER, models that have been trained with orders of magnitude more data and compute.