El Departamento de Nosotros: How Machine Translated Corpora Affects Language Models in MRC Tasks
Table 3: Performance on MLQA and XQuAD. Cased models. Best in bold, second best underlined.
['Model', 'MLQA EM', 'MLQA F1', 'XQuAD EM', 'XQuAD F1']
In the second experiment, we probe the TAR and MT fine-tuned models against MLQA and XQuAD in the Spanish context - Spanish question settings. A clear winner is BETO which is pre-trained on Spanish-only corpora and outperforms nearest contenders by about 2 F1 points. We then observe that TAR-trained models perform consistently better than MT-trained models in terms of F1 scores. Interestingly, in terms of EM scores mBERT 1.1 MT yields better performance than TAR and even language-specific models like BETO. Such a phenomena can be explained by robustness of large-scale multilingual LMs that might tend to generalize better over translation artifacts. We leave further research of this phenomena to the future work.