A BERT Baseline for the Natural Questions
Table 1: Our results on NQ compared to the baselines in the original dataset paper and to the performance of a single human annotator and of an ensemble of human annotators. The systems used in previous NQ baselines are DocumentQA Clark and Gardner (2017), DecAtt Parikh et al. (2016), and Document Reader Chen et al. (2017).
['[EMPTY]', 'Long Answer Dev P', 'Long Answer Dev R', 'Long Answer Dev F1', 'Long Answer Test P', 'Long Answer Test R', 'Long Answer Test F1', 'Short Answer Dev P', 'Short Answer Dev R', 'Short Answer Dev F1', 'Short Answer Test P', 'Short Answer Test R', 'Short Answer Test F1']
Our BERT model for NQ performs dramatically better than the models presented in the original NQ paper. Our model closes the gap between the F1 score achieved by the original baseline systems and the super-annotator upper bound by 30% for the long answer NQ task and by 50% for the short answer NQ task. However NQ appears to be still far from being solved, with more than 20 F1 points of headroom for both the long and short answer tasks.