Training Question Answering Models From Synthetic Data
Table 7: Effect of pretraining and scale on question filtration. Synthetic questions and answers were both generated with 1.2 billion parameter models. Before finetuning, overgeneration and filtration were performed with the models ablated here.
['Filter Model', '# Questions', 'EM', 'F1']
We then filter these questions with roundtrip filtration before finetuning a question answering model. Interestingly, these results follow an opposite trend compared to the previous section. In the previous section improvements to pretraining scale and tasks made a larger difference on answer generation than increasing model scale. However, here we see the opposite: improvements to pretraining tasks results only in a modest improvement to question filtration, while increasing model size results in much more substantive improvements. We hypothesize that this is due to the larger model’s ability to correctly answer more questions, and therefore allow more valid and high quality samples through to the finetuning phase as indicated by the number of questions generated by the technique.