Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
Table 6: The comparison with the previous semi-supervised QA method. All results are the performance on the full development set of SQuAD, i.e., our QA test + development set.
['Methods', 'New Data Size', 'EM', 'F1']
We conduct two ablation studies on the development set. First, we compare beam search with different beam sizes and diverse beam search Li et al. Second, increasing the size of synthetic data from H1 to H10, the performance saturates around H2-H4. We also observed that when using a big synthetic dataset, e.g., H10, the model converges even before all examples were used for training. Based on these results, we conjecture that there is an upper bound of the effect of synthetic data which might be limited by the QG quality. To further improve the performance, more diverse and tricky questions need to be generated. To show how QG models help or limit the QA performance, we include some synthetic QA examples in Appendix. Finally, we compare our semi-supervised QA methods with Dhingra et al.