Knowledge Enhanced Contextual Word Representations
Table 1: Comparison of masked LM perplexity, Wikidata probing MRR, and number of parameters (in millions) in the masked LM (word piece embeddings, transformer layers, and output layers), KAR, and entity embeddings for BERT and KnowBert. The table also includes the total time to run one forward and backward pass (in seconds) on a TITAN Xp GPU (12 GB RAM) for a batch of 32 sentence pairs with total length 80 word pieces. Due to memory constraints, the BERTLARGE batch is accumulated over two smaller batches.
['System', 'PPL', 'Wikidata MRR', '# params. masked LM', '# params. KAR', '# params. entity embed.', 'Fwd. / Bwd. time']
To rule out minor differences due to our data preparation, the BERT models are fine-tuned on our training data before being evaluated. Overall, KnowBert improves the masked LM perplexity, with all KnowBert models outperforming BERTLARGE, despite being derived from BERTBASE.