Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering
Table 4: A comparison of top results using the BERT-large model.
['Model Choice', 'SQuAD Test F1', 'SQuAD NER F1']
SQuAD F1 results reflect results on the hidden SQuAD test set. We report single-model numbers; \newcitelewis-etal-2019-unsupervised report an ensemble method achieving 56.40 F1 and a best single model achieving 54.7 F1. We make use of the whole-word-masking version of BERT-large, although using the original BERT-large gives similar performance of 62.69 on the SQuAD dev set. We report numbers on the sample of SQuAD questions which are named entities, which we refer to as SQuAD-NER. The subset corresponding to the SQuAD development dataset has 4,338 samples, and may differ slightly from Lewis et al. We also trained a fully-supervised model on the SQuAD training dataset with varying amounts of data and found our unsupervised performance equals the supervised performance trained on about 3,000 labeled examples.