Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia
Table 6: Performance of the neural machine reading comprehension model (no initialization with pretrained embeddings) on our generated corpus.
['[EMPTY]', 'Exact Match Dev', 'Exact Match Test', 'F-1 Dev', 'F-1 Test']
when applied to the development and test set portions of our generated dataset. The system was trained on the training set portion of our dataset. We use the SQuAD evaluation scripts, which calculate exact match (EM) and F-1 scores. We also train the DocReader on our training set and test the models’ performance on the original dev set of SQuAD; for this, the performance is around 45.2% on EM and 56.7% on F-1 metric. DocReader trained on the original SQuAD training set achieves 69.5% EM, 78.8% F-1 indicating that our dataset is more difficult and/or less natural than the crowd-sourced QA pairs of SQuAD.