SelQA: A New Benchmark for Selection-based Question Answering
TABLE XII: Accuracies on the SelQA evaluation set for answer triggering with respect to topics.
['[BOLD] Topic', '[BOLD] CNN0', '[BOLD] CNN2', '[BOLD] RNN1', '[BOLD] Q']
The accuracy is measured on the subset of questions that contain at least one answer among candidates; the top ranked sentence is taken and checked for the correct answer. Similar to answer sentence selection, CNN2 stills shows strength on topics such as ‘Country’ and ‘Historical Events’, but the trend is not as clear for the other models. The worst performing topics are ‘TV’, ‘Music’ and ‘Art’. Such a noticeable difference might be caused by the unusual semantic sentence constructions of the text. Sections in these categories often contain listings, bullet-pointed texts etc., which is problematic for the models to properly take care of. How to correctly understand and solve question from such context will be a challenge to the future systems. Also, interestingly, the standard deviation is much smaller for RNN1 (3.9%) compared to the CNN models (10-12%) although RNN1’s overall performance is lower.