LAReQA: Language-agnostic answer retrieval from a multilingual pool
Table 3: Performance on modified versions of XQuAD-R where one target answer is removed, either from the same language as the question (−same) or a random other language (−rand).
['[EMPTY]', 'mAP − rand', 'mAP − same', '% Δ', 'Rank (% Δ)']
We rerun the XQuAD-R evaluation, but for each question, we remove one of its 11 target answers from the multilingual candidate pool. If a model is free of same-language bias, the effect of removing a single target should be constant, regardless of whether the removed target was in the same language as the question or not. Looking at the delta between these conditions, the X-Y baseline only displays a minimal bias, falling from 0.62 to 0.61 mAP. The most affected model is X-X, whose training procedure actively encouraged same-language bias. Interestingly even En-En shows a significant delta, indicating that simply fine-tuning mBERT on English QA is not sufficient to produce an embedding space that is strongly language agnostic.