On the Cross-lingual Transferability of Monolingual Representations
Table 2: MLDoc and PAWS-X results (accuracy). mBERT results are from Eisenschlos et al. (2019) for MLDoc and from Yang et al. (2019) for PAWS-X, respectively. We bold the best result in each section with more than two models and underline the overall best result.
['[EMPTY]', '[EMPTY]', 'MLDoc en', 'MLDoc fr', 'MLDoc es', 'MLDoc de', 'MLDoc ru', 'MLDoc zh', 'MLDoc avg', 'PAWS-X en', 'PAWS-X fr', 'PAWS-X es', 'PAWS-X de', 'PAWS-X zh', 'PAWS-X avg']
In this task, we observe that simpler models tend to perform better, and the best overall results are from CLWE. We believe that this can be attributed to: (i) the superficial nature of the task itself, as a model can rely on a few keywords to identify the genre of an input document without requiring any high-level understanding and (ii) the small size of the training set. Nonetheless, all of the four model families obtain generally similar results, corroborating our previous findings that joint multilingual pre-training and a shared vocabulary are not needed to achieve good performance. Similar to experiments on other datasets, MonoTrans is competitive with the best joint variant, with a difference of only 0.6 points when we learn language-specific position embeddings.