Training Question Answering Models From Synthetic Data
Table 6: Comparison of answer generator pretraining and scale. Our 1.2 billion parameter question generator is used for generating questions.
['Answer Generator', '#Questions', 'EM', 'F1']
Answer generation is equally important in our data generation pipeline. Answer generation is the first component of the pipeline and must be precise to avoid compounding errors. For answer generation we use an unconditional extractive BERT model that predicts start and end spans jointly over a given sentence. From each probability distribution we sample the entire nucleus (p=0.9) or the top-5 spans, choosing whichever is smaller. To test the quality of the selected answers we generate questions from our 1.2 billion parameter question generator and finetune a question answering model on the synthesized questions without any filtration. We find that improvements in pretraining data and tasks dramatically improve answer generation quality by 1.4 EM and 0.3 F1 between BERT-Large and our 345 million parameter answer generation model. We find that increasing model scale further to 1.2 billion parameters improves answer generation quality F1 by 0.4 while EM only improves by 0.1.