Zero-shot Neural Retrieval viaDomain-targeted Synthetic Query Generation
Table 1: Zero-shot ad-hoc retrieval. Unsupervised∗; Out-of-domain†; Synthetic‡. Bold=Best; Underline=Best non-hybrid.
['[BOLD] Models', '[BOLD] Average', '[BOLD] Average  [BOLD] Prec', '[BOLD] Average  [BOLD] nDCG', '[BOLD] BioAsq', '[BOLD] BioAsq  [BOLD] Prec', '[BOLD] BioAsq  [BOLD] nDCG', '[BOLD] NaturalQuestions', '[BOLD] NaturalQuestions  [BOLD] Prec', '[BOLD] NaturalQuestions  [BOLD] nDCG', '[BOLD] Forum Travel', '[BOLD] Forum Travel  [BOLD] Prec', '[BOLD] Forum Travel  [BOLD] nDCG', '[BOLD] Forum Ubuntu', '[BOLD] Forum Ubuntu  [BOLD] Prec', '[BOLD] Forum Ubuntu  [BOLD] nDCG']
We report Mean Average Precision over the first 100 results (MAP), Precision@10 and nDCG@10. For NaturalQuestions we report Precision@1 since there is a single relevant document annotation. Here we compare the zero-shot systems on three data sets across four domains. First, on average, QGen is the best neural retrieval model. This suggests that synthetic query generation is a viable option for zero-shot retrieval, often outperforming using gold out-of-domain data and unsupervised neural models. However, the efficacy of the model depends on the specific domain. The more specialize the domain, the better the relative performance of QGen, specifically when compared to QA because QA contains general knowledge question-answer pairs, we would expect it to do well on datasets like NaturalQuestions, which also covers general knowledge questions. However, on highly specialized data like BioAsq, QGen is by far the most superior system. For BIOASQ and NQ we have corresponding in-domain training sets, though they are significantly smaller than weakly supervised sets used for QA and QGen. For BIOASQ, the supervised results are worse than the zero-shot models, however, for NQ, the supervised results are significantly better. Note NQ has a much larger supervised training set than BIOASQ, especially in terms of unique queries (70K vs 2.5K). The result suggests that even in domains with small supervised training sets, synthetic query generation is a powerful alternative.