Passage Re-ranking: A Comparative Analysis
Table 2. Average MRR and # of queries (in parenthesis) for ranges of FQT.
['[BOLD] FQT', '[0,0.1)', '[0.1,0.15)', '[0.15,0.2)', '[0.2,0.25)', '[0.25,1]']
To investigate this, we see how MRR changes across different ranges of FQT. As FQT value increases, we can see that the MRR value decreases in both BERT and BM25. Because of BM25’s bias towards high FQT (validated by Hypothesis I and also evident by the number of queries), we can see the decrease in MRR (as we go from the lowest to highest FQT buckets) is more prominent for BM25 (34.5%) than BERT (16.7%). The signed t-test for measuring the difference between two pairs of data, applied on difference between FQT values of BM25 and BERT yields a p-value of 0.0, indicating statistically significant difference between the FQT values.