Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge
Table 4: Injecting New Facts. In the Full setting the model is exposed to full knowledge in the pretraining data and KB. In the Filter setting, the models have access to no direct knowledge about question answer entity pairs from either the pretraining corpus or KB. In the Inject Facts setting, the pretraining corpus and training KB are still Filtered, but at inference time, new facts are injected into the models memory allowing it to recover most of the drop from the Full setting. In all cases, we remove the overlap between the finetune train and eval sets.
['[EMPTY]', 'FreebaseQA Full', 'FreebaseQA Filter', 'FreebaseQA Inject Facts', 'WebQuestionsSP Full', 'WebQuestionsSP Filter', 'WebQuestionsSP Inject Facts']
In the “Full” column, we pretrain and finetune the FaE model with the full knowledge base and corpus. In the “Filter” setting, facts about the finetuning data are hidden from the model at both pretraining and finetuning time. In this case, the model should fall back to the language model to predict the answer. In the “Inject Facts” setting, Facts are hidden at pretraining time, but are injected at test time. The results show that FaE can effectively use the newly injected facts to make prediction, i.e. an absolute improvement of 9.3% compared to the “Filter” setting.