LAReQA: Language-agnostic answer retrieval from a multilingual pool
Table 4: mAP on the zero-shot version of XQuAD-R, retrieving a single answer from a monolingual pool that matches the question language. Note, only En-En is strictly “zero-shot”, as the other models see machine translated fine-tuning data.
['[EMPTY]', '[BOLD] ar', '[BOLD] de', '[BOLD] el', '[BOLD] en', '[BOLD] es', '[BOLD] hi', '[BOLD] ru', '[BOLD] th', '[BOLD] tr', '[BOLD] vi', '[BOLD] zh', '[BOLD] Avg', 'angle=90,lap=0pt-(1em)Zero-shot  [BOLD] Rank', 'angle=90,lap=0pt-(1em)LAReQA  [BOLD] Rank']
Remarkably, the model ranking under this task diverges from that under our proposed LAReQA task of retrieval from a multilingual pool. In particular, the X-X(-mono) baselines which were only trained on “within-language” examples now perform the best, beating the top LAReQA baselines Translate-Test and X-Y. Figure embeddings of all XQuAD-R questions and candidates in English and Chinese (chosen as they are distant languages). The X-X embeddings show a dramatic separation between Chinese and English. This is a clear case of weak alignment, where a model achieves good zero-shot performance (cf. We believe there is significant headroom for models to improve on LAReQA. Our best initial baseline sidesteps the alignment problem by simply translating all test data to English. Among embedding-based models, our strongest baseline (“X-Y”) actively removes language bias by augmenting training data to include machine-translated cross-lingual examples. It is an interesting question for future work whether strong alignment always comes at a cost, or if better training techniques will lead to models that can improve on all these measures simultaneously.