Cross-Linguistic Syntactic Evaluation of Word Prediction Models
Таблица 7: Number of minimal sets used by BERT (English monolingual only) and mBERT for evaluation. The number of monolingual English examples is the same as in bertgoldberg19. Hyphens indicate non-replicable stimulus types, and 0 indicates that all focus verbs for a given stimulus type were out-of-vocabulary.
['[EMPTY]', 'English Mono', 'English Multi', 'French', 'German', 'Hebrew', 'Russian']
Important to note is that for these evaluations, we use stimulus types containing both animate and inanimate nouns to better match bertgoldberg19’s experimental setup; this is why we have more examples for English in this table than for the LSTM evaluations. Including or excluding inanimate nouns was found to make no significant difference in the final scores (for BERT or mBERT) regardless, since the performance of the model never diverges by more than 0.02 for animate vs. inanimate stimulus types.