On the Cross-lingual Transferability of Monolingual Representations
Table 4: Semantic and syntactic probing results of a monolingual model and monolingual models transferred to English. Results are on the Word-in-Context (WiC) dev set, the Stanford Contextual Word Similarity (SCWS) test set, and the syntactic evaluation (syn) test set Marvin and Linzen (2018). Metrics are accuracy (WiC), Spearman’s r (SCWS), and macro-averaged accuracy (syn).
['[EMPTY]', '[EMPTY]', 'mono en', 'xx→en aligned en', 'xx→en aligned fr', 'xx→en aligned es', 'xx→en aligned de', 'xx→en aligned el', 'xx→en aligned bg', 'xx→en aligned ru', 'xx→en aligned tr', 'xx→en aligned ar', 'xx→en aligned vi', 'xx→en aligned zh', 'xx→en aligned avg']
The results indicate that monolingual semantic representations learned from non-English languages transfer to English to a degree. On WiC, models transferred from non-English languages are comparable with models trained on English. On SCWS, while there are more variations, models trained on other languages still perform surprisingly well. In contrast, we observe larger gaps in the syntactic evaluation dataset. This suggests that transferring syntactic abstractions is more challenging than semantic abstractions. We leave a more thorough investigation of whether joint multilingual pre-training reduces to learning a lexical-level alignment for future work.