Simple Applications of BERT for Ad Hoc Document Retrieval
Table 2: Results on Robust04. FT indicates the dataset used for fine tuning; nS indicates inference using the top n scoring sentences of the document.
['Model', 'AP', 'P20']
The paper aggregation site “Papers With Code” places Lin’s result as the state of the art on Robust04 as of this writing. As a point of comparison, in the most recent survey of neural ranking models by Guo et al. , the best AP on Robust04 is in the 0.29 range, consistent with the above site. Therefore, we are quite confident that we are evaluating against competitive models. In the results table, “FT” indicates the dataset used for fine-tuning and nS indicates inference using the top n scoring sentences of the document. We find that the learned w4 value is zero, indicating that additional sentences do not help beyond the top three (at least according to our tuning procedure); thus, 4S results are omitted from the table. Interestingly, we find that fine-tuning BERT on microblog data is more effective than QA data, suggesting that task (QA vs. relevance matching) is more important than document genre (tweets vs. newswire). Cognizant of the potential dangers of repeated hypothesis testing, we probed the statistical significance of one five-fold setting, BM25+RM3 vs. “3S: BERT FT (Microblog)”. According to a paired t-test, the differences are statistically significant (p<10−7).