Passage Ranking with Weak Supervision
Table 3: PR performance on different datasets. SOTA: Cohen et al. (2018) for WikipassageQA, Rücklé & Gurevych (2017) for InsuranceQA_v2, Nogueira & Cho (2019) for MS-MARCO. The best numbers achieved by weak supervision models are in bold. ∗ indicates the new SOTA performance in full supervision.
['method', 'WikipassageQA (test) MAP', 'WikipassageQA (test) MRR', 'WikipassageQA (test) P@1', 'WikipassageQA (test) P@5', 'InsuranceQA_v2 (test) MAP', 'InsuranceQA_v2 (test) MRR', 'InsuranceQA_v2 (test) P@1', 'InsuranceQA_v2 (test) P@5', 'MS-MARCO (validation) MAP', 'MS-MARCO (validation) MRR', 'MS-MARCO (validation) P@1', 'MS-MARCO (validation) P@5']
After aggregating the results of labeling functions, we now train our BERT-PR model. We compare the final performances of different models with different supervision signals along with the unsupervised BM25 baseline. We use mean average precision (MAP), mean reciprocal rank (MRR), precision at 1 (P@1) and 5 (P@5) as our evaluation metrics.