Knowledge-Aware Language Model Pretraining
Table 4: Zero-shot question answering performance of different models for three different question answering benchmarks. EM is exact match percent, and cover-EM is the percent counting whether the correct answer is a substring of the generated answer. All generated answers were generated by a greedy decoding of at most 20 tokens.
['Model', 'Trivia QA EM', 'Trivia QA cover-EM', 'Natural Questions EM', 'Natural Questions cover-EM', 'Web Questions EM', 'Web Questions cover-EM']
Remarkably, even though the largest GPT-2 model was only trained on unsupervised text prediction tasks, it still performs nearly as well as the fully-supervised T5 11B model on Trivia QA, confirming that indeed, larger language models do capture some knowledge in an unsupervised way. Yet, the captured knowledge seems more on the “trivial side” as GPT-2 performs much worse on NQ and WQ, which are more similar to what real users ask in search engines. This observation still holds even in the newly-announced GPT-3 with 175 Billion parameters brown2020language, which has 30 EM on NQ, versus 44.5 in the RAG model which uses “only” hundreds of millions parameters lewis2020retrieval.