Passage Ranking with Weak Supervision
Table 2: Quality of labeling functions. BM25: BM25 scores for each pair. TF-IDF: TF-IDF similarity scores for each pair. BERT: cosine similarity scores over the feature extracted from the last layer of hidden state of pretrained BERT base model. Universal: cosine similarity scores over features from universal sentence embedding. MV: labels after majority voting. GM: predicted labels of learned generative model.
['pseudo label', 'WikipassageQA P@1', 'WikipassageQA R@1', 'WikipassageQA AUC', 'InsuranceQA_v2 P@1', 'InsuranceQA_v2 R@1', 'InsuranceQA_v2 AUC', 'MS-MARCO P@1', 'MS-MARCO R@1', 'MS-MARCO AUC']
We calculate cosine-similarity of BERT features and universal sentence embedding. To measure the quality of our labeling functions, we apply these labeling function on the training sets and compare our pseudo labels with the ground truth labels. Note that in ranking datasets, positive and negative pairs are highly imbalanced. So here we use precision and recall at 1 (P@1, R@1), and AUC to measure the quality of pseudo labels. We learn the simple generative model (GM) over labeling functions to estimate the true label. Also we show the result of majority voting strategy.