Training Question Answering Models From Synthetic Data
Table 10: Effect of filtration modeling choices on questions generated from ground truth and synthetic answers. 1.2 billion parameter models are used for every stage of the generation pipeline. Questions from no filtration are used in the other experiments with a second set of questions generated in overgeneration experiments.
['Filter Model', '# Questions', '345M QA EM', '345M QA F1', 'Large QA EM', 'Large QA F1']
Both question generation and answer generation sometimes produces poor answers. However, we find that even on questions generated from ground truth answers roundtrip filtering throws away questions associated with perfectly good answers. Throwing away data significantly hurts BERT- Large whose pretrained features are not as robust as our BERT-345M model and require more finetuning data. To combat this we take an approach similar to overgeneration and reranking ( We term this overgeneration and filtration. This helps avoid losing important answers in our synthesized training set. To perform overgeneration we sample one question with top-k (k=40) sampling and one with top-p (p=0.9) nucleus sampling. This leads approximately to a whole point of improvement for our model in both the case with and without ground truth answers, and allows us to surpass training with real SQuAD1.1 data.