Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Table 6: Ablation Studies on Masked Language Model and Masking Ratios.
['[BOLD] Model', '[BOLD] SQuAD EM', '[BOLD] SQuAD F1', '[BOLD] TriviaQA EM', '[BOLD] TriviaQA F1', '[BOLD] Quasar-T EM', '[BOLD] Quasar-T F1', '[BOLD] FIGER Acc']
The results of WKLM without MLM validate that adding the language model objective is essential for downstream performance. We also find that masking out too many words (i.e., 15% masking ratio as in the original BERT) leads to worse results. We conjecture that too many masked words outside entity mentions break parts of the context information and introduce noisy signals to knowledge learning. Results of continued BERT training show that more MLM updates are often beneficial, especially for SQuAD. However, on tasks that are more entity-centric, continued MLM training is less effective than our WKLM method. This suggests that our WKLM method could serve as an effective complementary recipe to masked language modeling when applied to entity-related NLP tasks.