Language Models with Transformers
Table 6: Compare model parameter size and results with GPT-2. The GPT-2 model size and results are from Radford et al. (2019).
['[BOLD] Model', '[BOLD] Parameters', '[BOLD] Datasets PTB', '[BOLD] Datasets WT-2', '[BOLD] Datasets WT-103']
We specifically compare the proposed model with the recent state-of-the-art language model GPT-2 Radford et al. WT-103. More surprisingly, the proposed method performs better than GPT-2 (1542M) which has around 4 times more parameters. On WT-103, BERT-Large-CAS is better than GPT-2 (762M) which has around 2 times more parameters. Note that on WT-2, our method performs worse than GPT-2, we suspect the reason is that the WebText still contains the texts that are similar to the Wikipedia. WT-2 is quite small in terms of scale. In contrast, we regard the results on WT-103 (50 times larger than WT-2) as a more reasonable comparison with GPT-2.