Measuring Domain Portability and Error Propagation in Biomedical QA
Table 6: Error propagation for factoid biomedical QA
['[EMPTY]', 'Gold Inputs google-gold-input', 'Gold Inputs google-gold-input', 'Gold Inputs google-gold-input', 'Noisy Inputs google-gold-pred', 'Noisy Inputs google-gold-pred', 'Noisy Inputs google-gold-pred', 'Δ', 'Δ', 'Δ']
We measure error propagation only for factoid QA for batches 3-5, which were the batches that we participated in. We can see from these results that feeding the QA model non-gold inputs leads to a dramatic drop in all metrics: from 7pts up to 14pts absolute. In one case (batch 5, strict accuracy), the metric is halved.