Contextualized Representations Using Textual Encyclopedic Knowledge
Table 4: F1 on TriviaQA and MRQA dev sets for different combinations of pretraining and finetuning.
['[EMPTY]', 'Pretraining', 'Finetuning', 'Wiki', 'Web', 'MRQA']
For these ablation studies, we use the 12-layer base models for all experiments, and report results, averaged over 3 random seeds. For models finetuned from the same Context-O checkpoint (rows 1 and 2), we see consistent improvements on TriviaQA (1.4 and 1.2 F1 for Wikipedia and Web respectively) and MRQA (0.2 F1) after enriching representations with encyclopedic knowledge. This underscores the advantage of textual encyclopedic knowledge in that it improves current models even without additional TEK-pretraining. The improvements from TEK-enriching task representations are even more prominent for models that have been TEK pretrained (rows 2 and 4) with gains of 2.2 F1 and 1.2 F1 on TriviaQA Wikipedia and Web respectively, and 0.4 on MRQA. We see that TEK pretraining shows consistent gains across both TriviaQA tasks (+0.9 and +0.4 F1) as well as MRQA (+0.3 F1). This shows that TEK-pretraining further improves the model’s ability to use the retrieved background knowledge to improve representations for the downstream RC task.