Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering
Table 6: F1 scores on 4 subsets of TriviaQA dev, grouped by the size of their answer string sets {\cal{A}} and corresponding set of possible mentions {\cal{I}}. \Delta indicates the improvement from H2-P to H3-D.
['Subset', '|\\mathcal{A}|', '|\\mathcal{I}|', 'Size', 'H2-P', 'H3-D', '\\Delta']
We categorize examples based on the size of their answer string set, |\mathcal{A}|, and the size of their corresponding set of {\cal{A}}-consistent spans, |\mathcal{I}|. In general, we expect \mathcal{Q}^{sl} and \mathcal{Q}^{ll} to be noisier due to the larger \mathcal{I}, where \mathcal{Q}^{sl} potentially includes many irrelevant mentions while \mathcal{Q}^{ll} likely contains more incorrect answer strings (false aliases). We can observe that the improvement is more significant for these noisier subsets, suggesting document-level modeling is crucial for handling both types of label noise.