BioNLP-OST 2019 RDoC Tasks: Multi-grain Neural Relevance Ranking Using Topics and Attention Based Query-Document-Sentence Interactions
Table 4: RDoC Task-1 analysis: Ranking of PubMed abstracts within “Potential Threat Anxiety (PTA)” cluster using supervised prediction probabilities (p(q|v)). It shows that an intruder/noisy abstract (Gold Label: Loss) is assigned higher probability than the abstracts with same Gold Label as the cluster. But, using re-ranking with BM25-Extra (reRank(BM25-Extra)) relevance score assigns lowest relevance to the intruder abstract.
['[BOLD] Ranking (with Prediction Probability)  [BOLD] PMID', '[BOLD] Ranking (with Prediction Probability)  [BOLD] Pred Prob', '[BOLD] Ranking (with Prediction Probability)  [BOLD] Gold Label', '[BOLD] Re-ranking (with BM25-Extra)  [BOLD] PMID', '[BOLD] Re-ranking (with BM25-Extra)  [BOLD] Gold Label']
“Loss”. When this cluster is ranked on the basis of prediction probabilities (p(q|v)), then “Loss” abstract is ranked third from the bottom and it degrades the mAP score of the retrieval system. But after re-ranking this cluster using reRank(BM25-Extra) relevance score, the “Loss” abstract is ranked at the bottom, thus maximizing the mAP score. Therefore, re-ranking with BM25-Extra on top of ranking with p(q|v) is, evidently, a robust abstract/document ranking technique.