Knowledge-Aware Language Model Pretraining
Table 3: Results on language modeling tasks and LAMA knowledge probing tasks. The LAMA numbers are average Precision@1.
['[EMPTY]', '[BOLD] Language Modeling Wiki-103', '[BOLD] Language Modeling Lambada', '[BOLD] LAMA Knowledge Probing G-Re', '[BOLD] LAMA Knowledge Probing T-REx', '[BOLD] LAMA Knowledge Probing T-REx', '[BOLD] LAMA Knowledge Probing T-REx', '[BOLD] LAMA Knowledge Probing T-REx', '[BOLD] LAMA Knowledge Probing C-Net', '[BOLD] LAMA Knowledge Probing Squad']
In general, the in-house GPT-2 results, with more training data, are slightly better than their Open AI implementations of equivalent architecture. We also consistently see that KALM has around 15%-20% improvement on LAMBADA accuracy. More importantly, on LAMA which directly tests factual correctness, KALM consistently improves over GPT-2 by margins of 40-80%, even approaching performance of a GPT-2 model with 5-fold more transformer parameters.