Language Models with Transformers
Table 1: Performance of Coordinate Architecture Search (CAS). ‘Val’ and ‘Test’ denote validation and test perplexity respectively.
['[BOLD] Model', '[BOLD] Datasets  [BOLD] PTB', '[BOLD] Datasets  [BOLD] PTB', '[BOLD] Datasets  [BOLD] WT-2', '[BOLD] Datasets  [BOLD] WT-2', '[BOLD] Datasets  [BOLD] WT-103', '[BOLD] Datasets  [BOLD] WT-103']
First note that GPT and BERT are significantly worse than AWD-LSTM-MoS. It confirms our hypothesis that neither BERT nor GPT are effective tools for language modeling. Applying them naively leads to significantly worse results compared to AWS-LSTM-MoS on three datasets. It demonstrates that language modeling requires strong capabilities in modeling the word order dependency within sentences. However, due to the combination of self-attention and positional encoding, both GPT and BERT primarily capture coarse-grained language representation but with limited word-level context.