How Much Knowledge Can You Pack Into the Parameters of a Language Model?
Table 1: Test set scores for T5 variants and previous results on the open-domain Natural Questions (NQ), WebQuestions (WQ), and TriviaQA (TQA) tasks.
['[EMPTY]', 'NQ', 'WQ', 'TQA']
Notably, performance on each dataset improves as the model size increases, with T5-11B performing best in every case. Further, we find that using Guu et al. ’s SSM pre-training produces a substantial boost in performance. T5-11B with SSM ultimately achieves state-of-the-art on both WebQuestions and TriviaQA beating all other methods except Guu et al. Importantly, all previous methods except Ling et al.