A Discrete Hard EM Approach for Weakly Supervised Question Answering
Table 3: Results on multi-mention reading comprehension & discrete reasoning tasks. We report performance on five datasets with different base models. Note that we are not able to obtain the test result on the subset DROPnum. Previous state-of-the-art are from Wang et al. (2018), Nishida et al. (2019), Lee et al. (2019), Lee et al. (2019) and Dua et al. (2019), respectively. Our training method consistently outperforms the First-Only and MML by a large margin in all the scenarios.
['[EMPTY]', 'TriviaQA', 'TriviaQA', 'NarrativeQA', 'NarrativeQA', 'TriviaQA -open', 'TriviaQA -open', 'NaturalQ -open', 'NaturalQ -open', 'DROPnum w/ BERT', 'DROPnum w/ QANet']
, we observe that First-Only is a strong baseline across all the datasets. We hypothesize that this is due to the bias in the dataset that answers are likely to appear earlier in the paragraph. Second, while MML achieves comparable result to the First-Only baseline, our learning method outperforms others by 2+ F1/ROUGE-L/EM consistently on all datasets. Lastly, our method achieves the new state-of-the-art on NarrativeQA, TriviaQA-open and NaturalQuestions-open, and is comparable to the state-of-the-art on TriviaQA, despite our aggressive truncation of documents.