Cross-Linguistic Syntactic Evaluation of Word Prediction Models
Таблица 3: English BERT (base) and multilingual BERT accuracies on the English stimuli from marvinlinzen18. Monolingual results are taken from bertgoldberg19.
['[EMPTY]', 'Mono', 'Multi']
Multilingual BERT performed better than English BERT on Sentential Complements, Short VP Coordination, and Across a Prepositional Phrase, but worse on Within an Object Relative Clause, Across an Object Relative Clause (no relative pronoun), and in Reflexive Anaphora Across a Relative Clause. The omission of the relative pronoun that caused a sharp drop in performance in mBERT, and a milder drop in English BERT. Otherwise, both models had similar accuracies on other stimuli. These results reinforce the finding in LSTMs that multilingual models generally underperform monolingual models of the same architecture, though there are specific contexts in which they can perform slightly better.