Contextualized Representations Using Textual Encyclopedic Knowledge
Table 5: Performance (F1) on TriviaQA and MRQA dev sets for varying lengths of context (NC) and background (NB). All models were finetuned from the same RoBERTa++ pretrained checkpoint.
['[ITALIC] NC', '[ITALIC] NB', 'Wiki', 'Web', 'MRQA']
Our approach incorporates textual knowledge by using a part of our Transformer window for it, instead of additional context from the same document. We first compare using a shorter window of 384 tokens for RC-contexts (question and passage) with using 512 tokens for RC-contexts (the first two rows). We see consistent gains from using longer document context, some of which our TEK-enriched representations need to sacrifice. We then consider the trade-off for varying values of context length NC and background length NB as indicated in rows 2-5. Our partitioning of 384 tokens for context and 128 for background outperforms other configurations. Moreover, adding up to 256 tokens of background knowledge improves performance over using only document context. This suggests that relevant encyclopedic knowledge about multiple entities from outside of the current document is more useful than long-distance neighboring text from the same document for these benchmarks.