Weaver: Deep Co-Encoding of Questions and Documents for Machine Reading
Table 6: Results when answering questions using the full English Wikipedia (top-1 EM accuracy, using SQuAD evaluation script)
['YodaQA', '[ITALIC] - addtl sources', 'SQuAD -', 'CuratedTREC 31.3', 'WebQuestions  [BOLD] 39.8', 'WikiMovies -']
Unlike recent machine reading comprehension datasets, CuratedTREC, WebQuestions and WikiMovies only contain question, answer pairs and lack any supporting documents. To gather supporting documents we resort to distant supervision but since we are interested in machine reading over long documents, we associate full documents instead of single paragraphs, and thus increase the character limit to 100,000 instead of 1,500 originally. We use those distantly supervised training sets to fine-tune a model trained on SQuAD. In addition to showing better performance than the original DrQA model on a single full document, Weaver is also able to handle more documents properly. While the EM score for DrQA reaches its maximum with 10 documents, we are able to increase the performance of the full system with Weaver as a reader by increasing the number of retrieved documents up to 25, see Fig. This allows to reach a final EM accuracy of 42.3 on SQuAD, which is more than 12 points better than the previous best reported performance.