SelQA: A New Benchmark for Selection-based Question Answering
TABLE IX: MRR scores on the SelQA evaluation set for answer sentence selection w.r.t. question types.
['[BOLD] Type', '[BOLD] CNN0', '[BOLD] CNN2', '[BOLD] RNN0', '[BOLD] RNN1', '[BOLD] Q']
The CNN models show strength on the ‘who’ type, whereas the RNN models show strength on the ‘when’ type. Each model varies on showing their weakness, which we will explore in the future. All models except for RNN0 tend to perform better as questions become longer. This makes sense since longer questions are usually more informative. On the other hand, models generally perform worse as sections become longer, which also makes sense because the models have to select the answer contexts from larger pools.