Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
Table 2: Zero-Shot (ZS) and Few-Shot (FS) comparison for Vanilla BERT (multilingual) on each dataset. Within each metric and dataset, the top result is listed in bold. Significant increases from using FS are indicated with ↑ (paired t-test, p<0.05).
['Dataset', 'P@20 ZS', 'P@20 FS', 'nDCG@20 ZS', 'nDCG@20 FS', 'MAP ZS', 'MAP FS']
To test whether a small amount of in-language training data can further improve BERT ranking performance, we conduct an experiment that uses the other collection for each language as additional training data. The in-language samples are interleaved into the English training samples. We find that the added topics for Arabic 2001 (+50) and Spanish 4 (+25) significantly improve the performance. This results in a model significantly better than BM25 for Arabic 2001, which suggests that there may be substantial distributional differences in the English TREC 2004 training and Arabic 2001 test collections. We further back this up by training an “oracle” BERT model (training on the test data) for Arabic 2001, which yields a model substantially better (P@20=0.7340, nDCG@20=0.8093, MAP=0.4250).