Simple and Effective Semi-Supervised Question Answering
Table 2: A holistic view of the performance of our system compared against baseline systems on SQuAD and TriviaQA. Column groups represent different fractions of the training set used for training.
['[BOLD] Model', '[BOLD] Method', '[BOLD] 0  [BOLD] F1', '[BOLD] 0  [BOLD] EM', '[BOLD] 0.01  [BOLD] F1', '[BOLD] 0.01  [BOLD] EM', '[BOLD] 0.05  [BOLD] F1', '[BOLD] 0.05  [BOLD] EM', '[BOLD] 0.1  [BOLD] F1', '[BOLD] 0.1  [BOLD] EM', '[BOLD] 0.2  [BOLD] F1', '[BOLD] 0.2  [BOLD] EM', '[BOLD] 0.5  [BOLD] F1', '[BOLD] 0.5  [BOLD] EM', '[BOLD] 0.9  [BOLD] F1', '[BOLD] 0.9  [BOLD] EM', '[BOLD] 1  [BOLD] F1', '[BOLD] 1  [BOLD] EM']
Deep learning systems have shown a lot of promise for extractive Question Answering (QA), with performance comparable to humans when large scale data is available. However, practitioners looking to build QA systems for specific applications may not have the resources to collect tens of thousands of questions on corpora of their choice. We apply our system to three datasets from different domains. SQuAD We also report the performance on the dev set when trained on the full training set ( We use the same hyperparameter settings as in prior work. We compare and study four different settings: 1) the Supervised Learning (SL) setting, which is only trained on the supervised data, 2) the best performing GDAN model from Yang et al. The LM and Cloze methods use exactly the same data for pretraining, but differ in the loss functions used. We report F1 and EM scores on our test set using the official evaluation scripts provided by the authors of the dataset. Without any fine-tuning (column 0) the performance is low, probably because the model never saw a real question, but we see significant gains with Cloze pretraining even with very little labeled data. The BiDAF+SA model, exceeds an F1 score of 50% with only 1% of the training data (454 questions for SQuAD, and 746 questions for TriviaQA), and approaches 90% of the best performance with only 10% labeled data. The gains over the SL setting, however, diminish as the size of the labeled set increases and are small when the full dataset is available.