Training Question Answering Models From Synthetic Data
Table 8: Effect of question generator modeling choices. Questions are generated from ground truth answers without any filtration.
['Question Generator', '# Questions', 'EM', 'F1']
To study question generation in isolation we used our 345 million parameter model to generate questions from ground truth SQuAD1.1 answers. We first investigated the use of pretrained models and found that pretraining our GPT-2 model was crucial for achieving reasonable question generation. We then examined the effect of stopword filtration in our question generator. We found that this provided a substantial boost to EM and F1 scores of 5.2 and 4.1 respectively. The goal of employing this technique was to catch generations that ramble onwards without stopping, or produce end of text prematurely in the middle of a question. On manual inspection we found qualitatively that this technique helped when generating questions on text that featured heavy use of symbols and foreign language. In these cases the model struggled with out of distribution vocabulary, autoregressive sampling degenerated, and no stopword was produced.