Coreferential Reasoning Learningfor Language Representation
Table 7: Ablation study on various benchmark datasets (F1).
['Model', 'QUOREF', 'SQuAD', 'NewQA', 'TriviaQA', 'SearchQA', 'HotpotQA', 'NaturalQA', 'DocRED']
In this subsection, we explore the effects of the Whole Word Masking (WWM), Mention Reference Masking (MRM), Next Sentence Prediction (NSP) and copy-based training objective using several benchmark datasets. We continue to train Google’s released BERTBASE on the same Wikipedia corpus with different strategies. The conclusion is consistent with Joshi et al. (2) MRM scheme usually achieves parity with WWM scheme except on SearchQA, and both of them outperform the original subword masking scheme on NewsQA (averagely +1.7% F1) and TriviaQA (averagely +1.5% F1); (3) On the basis of mention reference masking scheme, our copy-based training objective explicitly requires model to look for noun ’s referents in the context, which could effectively consider the coreference information of the sequence. CorefBERT takes advantage of the objective and further improves performance, with a substantial gain (+2.3% F1) on QUOREF.