Training Question Answering Models From Synthetic Data
Table 9: Effect of answer generator modeling choices. Model based answer generation is performed with BERT-345M and questions are generated using a 1.2B parameter model. No filtration is applied to the generated questions.
['Answer Generator', '# Questions', 'EM', 'F1']
In our experiments we found answer generation to be a significant bottleneck in performance. However, these scaling analysis were performed with our best model. Starting with a Named Entity Recognition (NER) model we find that it gets a dismal EM and F1 score. This is due to entities comprising only of ∼ 50% of the answer distribution for SQuAD1.1 . It’s necessary to use a learned model to model the diverse set of answers present SQuAD1.1 . We then tried to use the most common SQuAD1.1 model, which models the start and end of a span independently, to extract answers from individual sentences. This performed noticeably better, boosting our score to 77.2 EM, despite producing fewer answers than NER extraction. However, upon inspection we found that modeling the span independently resulted in sampling repetitive answers. This is the model we ended up choosing as it performed the best with an exact match score of 79.1. Lastly, we also considered jointly modeling answer spans over an entire paragraph instead of a single sentence. However, we found that it performed worse than independent span modeling over sentences.