PACRR: A Position-Aware Neural IR Model for Relevance Matching
Table 1: ERR@20 and nDCG@20 on Trec Web Track 2012–14 when re-ranking search results from QL. The comparisons are conducted between two variants of PACRR and DRMM (D/d), DUETL (L/l), MatchPyramid (M/m) and K-NRM (K/k). All methods are compared against the QL (Q/q) baseline. The upper/lower-case characters in the brackets indicate a significant difference under two-tailed paired Student’s t-tests at 95% or 90% confidence levels relative to the corresponding approach. In addition, the relative ranks among all runs within the respective years according to ERR@20 and nDCG@20 are also reported directly after the absolute scores.
['Measure  [ITALIC] ERR@20', 'Years wt12', 'PACRR-firstk 0.318\xa0( [ITALIC] mQ)', 'Rank 2', 'PACRR-kwindow 0.313\xa0( [ITALIC] MQ)', 'Rank 4', 'DUETL 0.281\xa0( [ITALIC] Q)', 'Rank 10', 'DRMM 0.289\xa0( [ITALIC] Q)', 'Rank 10', 'MatchPyramid 0.227', 'Rank 16', 'K-NRM 0.258\xa0( [ITALIC] Q)', 'Rank 12', 'QL 0.177', 'Rank 26']
RerankSimple. We first examine the proposed model by re-ranking the search results from the QL baseline on Web Track 2012–14. It can be seen that DRMM can significantly improve QL on WT12 and WT14, whereas MatchPyramid fails on WT12 under ERR@20. While DUETL and K-NRM can consistently outperform QL, the two variants of PACRR are the only models that can achieve significant improvements at a 95% significance level on all years under both ERR@20 and nDCG@20. More remarkably, by solely re-ranking the search results from QL, PACRR-firstk can already rank within the top-3 participating systems on all three years as measured by both ERR and nDCG. The re-ranked search results from PACRR-kwindow also ranks within the top-5 based on nDCG@20. On average, both PACRR-kwindow and PACRR-firstk achieve 60% improvements over QL.