A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction
Table 3: Results on three multiple-choice reading comprehension datasets. (F1a: F1 score on all answer-options; F1m: macro-average F1 score of all questions; EM0: exact match.) Note that there is no golden evidence label on RACE and DREAM. The results for DPL (deep programming logic) are copied from (Wang et al., 2019). Significance tests were conducted between BERT-HA+STM and the best baseline of each column (t-test). ‡ means p-value <0.01, and † means p-value <0.05.
['Model / Dataset', 'RACE-M Dev', 'RACE-M Test', 'RACE-H Dev', 'RACE-H Test', 'MultiRC Dev', 'MultiRC Dev', 'MultiRC Dev', 'DREAM Dev', 'DREAM Test']
We adopt the metrics from the referred papers. STM improved BERT-HA consistently on RACE-H, MultiRC and DREAM in terms of all the metrics. However, the improvement on RACE-M is limited (1.0 gain on the test sets). The reason may be that RACE-M is much simpler than RACE-H, and thus, it is not challenging for the evidence extractor of BERT-HA to find the correct evidence on RACE-M.