Cross-Linguistic Syntactic Evaluation of Word Prediction Models
Таблица 2: Multilingual BERT accuracies on CLAMS. If a hyphen is present, this means that all focus verbs for that particular language and construction were out-of-vocabulary. Chance accuracy is 0.5.
['[EMPTY]', 'English', 'French', 'German', 'Hebrew', 'Russian']
mBERT’s accuracies on all stimuli. Performance on CLAMS was fairly high in the languages that are written in Latin script (English, French and German). On English in particular, accuracy was high across conditions, ranging between 0.83 and 0.88 for sentences with relative clauses, and between 0.92 and 1.00 for the remaining conditions. Accuracy in German was also high: above 0.90 on all constructions except Across Subject Relative Clause, where it was 0.73. French accuracy was more variable: high for most conditions, but low for Across Subject Relative Clause and Across Prepositional Phrase. We see the opposite trend for mBERT In other words, accuracy is likely to decrease with increasing morphological complexity. This unexpected inverse correlation may be an artifact of mBERT’s limited vocabulary, especially in non-Latin scripts. Morphologically complex languages have more unique word types. In some languages, this issue can be mitigated to some extent by splitting the word into subword units, as BERT does; however, the effectiveness of such a strategy would be limited at best in a language with non-concatenative morphology such as Hebrew. Finally, we stress that the exclusion of certain stimulus types and the differing amount of training data per-language act as confounding variables, rendering a comparison between mBERT and LSTMs difficult.