SelQA: A New Benchmark for Selection-based Question Answering
TABLE VII: MRR scores on the SelQA evaluation set for answer sentence selection with respect to topics.
['[BOLD] Topic', '[BOLD] CNN0', '[BOLD] CNN2', '[BOLD] RNN0', '[BOLD] RNN1', '[BOLD] Q']
All models show strength on topics such as ‘Country’ and ‘Historical Events’, which is comprehensible since questions in these topics tend to be deterministic. On the other hand, most models show weakness on topics such as ‘TV’, ‘Arts’, or ‘Music’. This may be due to the fact that not many overlapping words are found between question and answer pairs in these documents, which also consist of many segments caused by bullet points.