Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Table 1: Zero-Shot Fact Completion Results.
['Relation Name', '# of Candidates', '# of Answers', 'Model  [BOLD] BERT-base', 'Model  [BOLD] BERT-large', 'Model  [BOLD] GPT-2', 'Model  [BOLD] Ours']
We denote our method WKLM for (Weakly Supervised Knowledge-Pretrained Language Model). Overall, WKLM achieves the best results on 8 of the 10 relations. We also observe that GPT-2 outperforms BERT on average. We think this is because the fact completion task requires models to predict the missing entities using only a short context on the left, while BERT pretraining incorporates context from both directions. Interestingly, BERT achieves good performance on several geographical relations such as PlaceOfBirth, LocatedIn and PlaceOfDeath. We conjecture that this is because location entities usually appear at sentence ends in Wikipedia articles , e.g., Obama was born in Honolulu, Hawaii.. This sentence pattern is similar to our templates and BERT may learn to rely mostly on the left context to make predictions. For most relations that include answers that are person names, BERT lags behind both GPT-2 and our model.