Multilingual Alignment of Contextual Word Representations
Table 2: Zero-shot accuracy on the XNLI test set, where we align BERT with varying amounts of parallel data. The method scales with the amount of data but achieves a large fraction of the gains with 50K sentences per language pair.
['Sentences', 'English', 'Bulgarian', 'German', 'Greek', 'Spanish', 'French', 'Average']
Given that our goal is zero-shot transfer, we cannot expect to always have large amounts of parallel data. We find that it improves transfer with as little as 10K sentences per language, making it a promising approach for low-resource languages.