Medical Exam Question Answering with Large-scale Reading Comprehension
Table 4: Test performance with different number of documents given per candidate answer
['Number of documents', 'top-1', 'top-5', 'top-10', 'top-20']
To discover the general relevancy of documents returned by our retrieval system, we hand-labeled the relevancy of 1000 retrieved documents for 100 problems. We notice that there is still performance gain using as many as 20 documents per candidate answer, while the ratio of relevant documents is already low. This illustrates SeaReader’s ability to discern useful information from large-scale text and integrate them in reasoning and decision-making.