Efficient and Robust Question Answering from Minimal Context over Documents
Table 8: Results on the dev-full set of TriviaQA (Wikipedia) and the dev set of SQuAD-Open. Full results (including the dev-verified set on TriviaQA) are in Appendix C. For training Full and Minimal on TriviaQA, we use 10 paragraphs and 20 sentences, respectively. For training Full and Minimal on SQuAD-Open, we use 20 paragraphs and 20 sentences, respectively. For evaluating Full and Minimal, we use 40 paragraphs and 5-20 sentences, respectively. ‘n sent’ indicates the number of sentences used during inference. ‘Acc’ indicates accuracy of whether answer text is contained in selected context. ‘Sp’ indicates inference speed. We compare with the results from the sentences selected by TF-IDF method and our selector (Dyn). We also compare with published Rank1-3 models. For TriviaQA(Wikipedia), they are Neural Casecades Swayamdipta et al. (2018), Reading Twice for Natural Language Understanding Weissenborn (2017) and Mnemonic Reader Hu et al. (2017). For SQuAD-Open, they are DrQA Chen et al. (2017) (Multitask), R3 Wang et al. (2018) and DrQA (Plain).
['[EMPTY]', '[EMPTY]', 'TriviaQA (Wikipedia) n sent', 'TriviaQA (Wikipedia) Acc', 'TriviaQA (Wikipedia) Sp', 'TriviaQA (Wikipedia) F1', 'TriviaQA (Wikipedia) EM', 'SQuAD-Open n sent', 'SQuAD-Open Acc', 'SQuAD-Open Sp', 'SQuAD-Open F1', 'SQuAD-Open EM']
First, Minimal obtains higher F1 and EM over Full, with the inference speedup of up to 13.8×. Second, the model with our sentence selector with Dyn achieves higher F1 and EM over the model with TF-IDF selector. For example, on the development-full set, with 5 sentences per question on average, the model with Dyn achieves 59.5 F1 while the model with TF-IDF method achieves 51.9 F1. Third, we outperforms the published state-of-the-art on both dataset.