Multilingual Alignment of Contextual Word Representations
Table 4: Accuracy by part-of-speech tag for non-contextual word retrieval. To achieve better word type coverage, we do not remove word pairs seen in the training set. The tags are grouped into lexically overlapping, closed-class, and open-class groups. The “Particle,” “Symbol,” “Interjection,” and “Other” tags are omitted.
['Lexical Overlap Base BERT', 'Numeral 0.90', 'Punctuation 0.88', 'Proper Noun 0.80', '[EMPTY]', '[EMPTY]', 'Average 0.86']
Next, to gain insight into the multilingual pre-training procedure, we analyze the accuracy broken down by part-of-speech using the Universal Part-of-Speech Tagset (petrov-etal-2012-universal), annotated using polyglot ( polyglot:2013:ACL-CoNLL) for Bulgarian and spaCy Unsurprisingly, multilingual BERT has high alignment out-of-the-box for groups with high lexical overlap, e.g. numerals, punctuation, and proper nouns, due to its shared vocabulary. We further divide the remaining tags into closed-class and open-class, where closed-class parts-of-speech correspond to fixed sets of words serving grammatical functions (e.g. determiner, preposition, conjunction, pronoun, and auxiliary), while open-class parts-of-speech correspond to lexical words (e.g. noun, adverb, adjective, verb). Interestingly, we see that base BERT has consistently lower accuracy for closed-class versus open-class categories (0.70 vs 0.54), but that this discrepancy disappears after alignment (0.88 vs 0.89).