Cross-Linguistic Syntactic Evaluation of Word Prediction Models
Таблица 1: LSTM LM test perplexities and accuracies on CLAMS across languages for the language-specific monolingual models and for our multilingual model. Results are averaged across five random initializations. Chance accuracy is 0.5. Boldfaced numbers indicate the model that achieved the highest performance on a given construction across languages.
['[EMPTY]', 'English Mono', 'English Multi', 'French Mono', 'French Multi', 'German Mono', 'German Multi', 'Hebrew Mono', 'Hebrew Multi', 'Russian Mono', 'Russian Multi']
Accuracy on short dependencies without attractors—Simple Agreement and Short VP Coordination—was close to perfect in all languages. This suggests that all monolingual models learned the basic facts of agreement, and were able to apply them to the vocabulary items in our materials. At the other end of the spectrum, performance was only slightly higher than chance in the Across an Object Relative Clause condition for all languages except German, suggesting that LSTMs tend to struggle with center embedding—that is, when a subject-verb dependency is nested within another dependency of the same kind marvinlinzen18; noji2020analysis.