\taskname: Answering Ambiguous Open-domain Questions
Table 7: Exact Match (EM) on NQ-open of different models, counting a prediction as correct if it matches Any gold reference, or only the First non-null one.
['Model', 'Any dev', 'Any test', 'First dev', 'First test']
As a consequence, there is a significant gap in model performance between development and test when they are evaluated under the same conditions. The official evaluation protocol for NQ-open counts a prediction as correct if it matches any of the gold reference answers. Under these conditions, the gap between development and test appears marginal However, as the NQ-open test set was more comprehensively annotated than development, it has a more generous evaluation; the number of unique reference answers is 1.2 and 1.8 on development and test, respectively.