MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Table 5: EM on the development set, where each row uses the same question with a different context, and AllContexts is a union of the other 3 datasets.
['[EMPTY]', 'TQA-G', 'TQA-U', 'TQA-W']
In the first 3 rows we train on 75K examples from each dataset, and in the last we train on the combined 225K examples. First, we observe that context augmentation substantially improves performance (especially for TQA-G and TQA-W). Second, generalization is sensitive to the context type: performance substantially drops when training on one context type and evaluating on another (60.7→ 48.4 for TQA-G, 53.1→44.6 for TQA-U, and 50.1→43.3 for TQA-W).