KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Table 10: Ablation study on TACRED (%). RoBERTa∗ and KEPLER-KE indicate only using the MLM and the KE objective, respectively.
['Model', 'P', 'R', 'F-1']
To demonstrate the effectiveness of the joint objective, we carry out experiments on training with only the MLM loss (RoBERTa∗) and only the KE loss (KEPLER-KE), and evaluate them on TACRED. It suggests that the performance gain of KEPLER is credit to the joint training towards both objectives, which enhances KEPLER with factual knowledge while keeping the strong language understanding ability.