Training Question Answering Models From Synthetic Data
Table 5: Effect of question generator scale on SQuAD1.1 performance. Ground truth answers are used to generate questions without filtration for finetuning.
['Question Generator', '# Questions', 'EM', 'F1']
Question generation plays a critical role in our synthetic data pipeline: it must synthesize linguistically and logically coherent text even if the text does not exist within the provided context. In this section we investigate the relationship between question generator scale and downstream SQuAD1.1 performance. We isolate the quality of question generation by using ground truth answers from the SQuAD1.1 dataset to generate questions and finetune a BERT model before evaluating it on the SQuAD1.1 dev set. We perform no question filtration in between generation and finetuning. Additionally, the number of valid samples that pass stopword filtration increase with larger models, indicating bigger models maintain coherency during sampling. For comparisons with prior work we train a question answering model with our ALBERT-style BERT model (BERT-345M) and the original BERT-Large model. Compared to our work we find that a similarly parameterized set of models achieve equal if not better performance despite using only a single supervised pass through the data and no feedback loop.