Multilingual Alignment of Contextual Word Representations
Table 5: Zero-shot accuracy on the XNLI test set with more languages, where we use 20K parallel sentences for each language paired with English. This result confirms that the alignment method works for distant languages and a variety of parallel corpora, including Europarl, MultiUN, and Tanzil, which contains sentences from the Quran (koehn-2005-europarl; eisele-chen-2010-multiun; tiedemann-2012-parallel).
['[EMPTY]', 'English', 'Bulgarian', 'German', 'Greek', 'Spanish', 'French', 'Arabic', 'Chinese', 'Urdu', 'Average']
] Figure 2: XNLI zero-shot versus word retrieval accuracy for base BERT, where each point is a language paired with English. This plot suggests that alignment correlates well with cross-lingual transfer.\ffigbox [1.1] Figure 3: Contextual word retrieval accuracy plotted against difference in frequency rank between source and target. The accuracy of base BERT plummets for larger differences, suggesting that its alignment depends on word pairs having similar usage statistics. From this closed-class vs open-class difference, we hypothesize that BERT’s alignment of a particular word pair is influenced by the similarity of their usage statistics. Specifically, given that BERT is trained through masked word prediction, its embeddings are in large part determined by the co-occurrences between words. Therefore, two words that are used in similar contexts should be better aligned. This hypothesis provides an explanation of the closed-class vs open-class difference: closed-class words are typically grammatical, so they are used in similar ways across typologically similar languages. Furthermore, these words cannot be substituted for one another due to their grammatical function. Therefore, their usage statistics are a strong signature that can be used for alignment. On the other hand, open-class words can be substituted for one another: for example, in most sentences, the noun tokens could be replaced by a wide range of semantically dissimilar nouns with the sentence remaining syntactically well-formed. By this effect, many nouns have similar co-occurrences, making them difficult to align through masked word prediction alone. We see that accuracy drops off significantly as the source-target difference increases, supporting our hypothesis. Furthermore, this shortcoming is remedied by alignment, revealing another systematic deficiency of multilingual pre-training.