LAReQA: Language-agnostic answer retrieval from a multilingual pool
Table 2: Mean average precision (mAP) of baseline models on XQuAD-R and MLQA-R.
['[EMPTY]', 'XQuAD-R', 'MLQA-R']
On both XQuAD-R and MLQA-R, the strongest model is the Translate-Test baseline. This is unsurprising in that LAReQA demands language-agnostic retrieval, and Translate-Test leverages an external machine translation system to actively “remove” the effects of language, by translating all test data to English. This result supports our claim that LAReQA tests for cross-lingual alignment in a way that existing zero-shot evaluations do not. As an extreme case, X-X scored a mere 0.23 on LAReQA mAP, compared to the best embedding model X-Y and the best overall baseline Translate-Test with 0.63 and 0.70 respectively. More generally, we observe that the more a model separates languages in embedding space, the worse it performs on LAReQA (cf. This ordering is also reflected in the degree to which language ID is predictable from the embeddings. When we use a linear regression to predict language (English vs. Chinese) from the question and candidate embeddings, the accuracies on a one-third holdout are X-X: 99.2%, En-En: 97.7%, X-X-mono: 87.5% and X-Y: 54.0%. This supports the claim that LAReQA is a better test of strong alignment than current zero-shot tasks.