Multilingual Alignment of Contextual Word Representations
Table 3: Word retrieval accuracy for the aligned sentence-augmented fastText baseline and BERT pre- and post-alignment. Across languages, base BERT has variable accuracy while fine-tuning-aligned BERT is consistently effective. Fine-tuned BERT also matches fastText in a version of the task where context is not necessary, suggesting that our method matches the type-level alignment of fastText while also aligning context.
['[EMPTY]', 'bg-en', 'de-en', 'el-en', 'es-en', 'fr-en', 'Average']
In the following sections, we present word retrieval results to both compare our method to past work and better understand the strengths and weaknesses of multilingual BERT. First, we find that in contextual retrieval, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This result supports the intuition that aligning large pre-trained models is more difficult than aligning word vectors, given that a rotation, at least when applied naively, produces sub-par alignments. In addition, fine-tuned BERT matches the performance of fastText in non-contextual retrieval, suggesting that our alignment procedure overcomes these challenges and achieves type-level alignment that matches non-contextual approaches. In the appendix, we also provide examples of aligned BERT disambiguating between different meanings of a word, giving qualitative evidence of the benefit of context alignment.