Quizbowl: The Case for Incremental Question Answering
Table 6: Best model is bolded. Each model is run five times with different random initializations. For the accuracy metrics we show the top score, mean score, standard deviation of scores. Expected win scores and training time are from the top model. The rnn models achieve the best results, but compared to dan models take significantly longer to train for 2.3% absolute improvement. Using Wikipedia as training data often helps, but is not a universal trend.
['[EMPTY]', '[EMPTY]', 'Accuracy (%) Start', 'Accuracy (%) Start', 'Accuracy (%) End', 'Accuracy (%) End', 'Expected Wins', 'Expected Wins', 'Time']
Although Wikipedia text is of a different distribution than questions, using it as additional training data is better than too little training data for specific answers. For example, there are zero training questions about Rocket, but a test question refers to Konstantin Tsiolkovksy’s development of rocket theory. However, this fact is mentioned on the Wikipedia page on rockets so with Wikipedia as training data answering this question is plausible. Each Quizbowl answer is paired with a Wikipedia page so we use the text of that page as additional training data. This section shows how we incorporate Wikipedia as training data in each model. We report an our primary version of ew (eager), and an alternative version of ew (stable). All models struggle with answering at the start of the question with the best score achieving only about 11% accuracy. This is unsurprising as the first sentence contains the most difficult clue, and they are difficult for even the best human players. The story is quite different at the end of the question since nearly all questions have giveaway clues. These clues are specifically designed to make the question extremely easy to answer so accuracy here should be near perfect. Despite this the best model achieves only 61% accuracy so there is much room for improvement. The ew scores for all models fall in between, but trend closer towards end accuracies. The difference in eager versus stable ew scores indicates that many although models can answer early correctly they often change their answer.