Multilingual Alignment of Contextual Word Representations
Table 1: Accuracy on the XNLI test set, where we compare to base BERT (devlin_bert:2018) and two rotation-based methods, sentence alignment (aldarmaki-diab-2019-context) and word alignment (wang-etal-2019-cross). We also include the current state-of-the-art zero-shot achieved by XLM (lample_xlm_2019). Rotation-based methods provide small gains on some languages but not others. On the other hand, after fine-tuning-based alignment, Bulgarian and Greek match the translate-train ceiling, while German, Spanish, and French close roughly one-third of the gap.
['[EMPTY]', 'English', 'Bulgarian', 'German', 'Greek', 'Spanish', 'French', 'Average']
We see that our alignment procedure greatly improves accuracies, with all languages seeing a gain of at least 1%. In particular, the Bulgarian and Greek zero-shot numbers are boosted by almost 5% each and match the translate-train numbers, suggesting that the alignment procedure is especially effective for languages that are initially difficult for BERT. We also run alignment for more distant language pairs (Chinese, Arabic, Urdu) and find similar results, which we report in the appendix. We also find that before alignment, BERT’s performance varies greatly between languages, while after alignment it is consistently effective. In particular, Bulgarian and Greek initially have very low accuracies.