Meta Answering for Machine Reading
Table 4: Results on the Short Answer task of the Natural Questions dataset.
['……………… System', 'Short Answer Dev P', 'Short Answer Dev R', 'Short Answer Dev Span F1', 'Short Answer Test P', 'Short Answer Test R', 'Short Answer Test Span F1']
We take as reference Alberti et al. ’s QA model trained on NQ and refer to it as BERT. Adding whole word masking The baseline, MMABase, is worse than BERTWWM. This shows that a naïve approach to encode evidential information is not comparable with the original answer context. The more careful handling of evidence, embeddings, and multiple losses lead the full MMA systems to outperforming MMABase by more than 6 F1 points on test. They also improve over BERTWWM by about 2 F1 points. This is counter intuitive, from a human perspective, as humans are better with more information. Together with the additional finding that this is flipped on dev—MMA\textscContext has higher precision and F1 than MMA\textscAnswerOnly—indicates that more work is needed to properly model context.