Meta Answering for Machine Reading
Table 1: A boostrap evaluation to fairly compare original NQ annotators, human meta-answerers, and the baseline QA system. In the context with the richest interaction, the best human meta-answerer can improve over both the average NQ annotator and the baseline QA system, mostly through improving recall.
['[EMPTY]', 'AnswerOnly Pr.', 'AnswerOnly Rec.', 'AnswerOnly F1', 'Context Pr.', 'Context Rec.', 'Context F1', 'RewriteQues Pr.', 'RewriteQues Rec.', 'RewriteQues F1']
Rather than rely on a deficient gold set, bootstrap sample from the remaining four annotations to get back to five annotations. But it would unfair to only evaluate NQ annotators using this, hence we evaluate our humans and the baseline bert QA system using the same metric (a bootstrap sample to get five annotations). This is a fair comparison between the NQ annotator, our human annotators, and the baseline QA system. All humans improve between AnswerOnly and Context. The better human meta-answerers improve over the baseline system. The best human (consistent across all conditions) was −7.9 worse on F1 in the AnswerOnly, the best human meta-answerer has a higher F1 than the average NQ annotator in RewriteQues with substantially less information.