Studio Ousia’s Quiz Bowl Question Answering System
Table 3: Accuracies of our question answering system. NQS and NTP stand for Neural Quiz Solver and Neural Type Predictor, respectively.
['Name', 'Sent 1', 'Sent 1–2', 'Sent 1–3', 'Full']
Here, we tested the performance using Dataset QA, and used the output of the Answer Scorer to predict the answer. Our system performed very accurately; it achieved 56% accuracy when given only a single sentence and 97% accuracy given the full set of sentences. To further evaluate the effectiveness of each sub-model presented above, we added the sub-models incrementally to the Answer Scorer. Note that the features not based on sub-models (e.g., the number of words in a question) were included in all instances. As a result, all of the sub-models effectively contributed to the performance. We also observed that the neural network models (i.e., Neural Quiz Solver and Neural Type Predictor) achieved good performance only for longer questions. Further, the IR models substantially improved the performance, especially for shorter questions.