Language Modeling with Deep Transformers
Table 4: Effect of activation functions. Perplexity after 1 epoch (10 sub-epochs in our setup) for (24, 2048, 512, 8).
['Activation', 'Perplexity Train', 'Perplexity Dev']
16 heads which is the largest number we try in this setup give the best performance. As opposed to previous work on feed-forward language models using GLUs [ As we observe that the impact of choice of activation functions on the perplexity is overall limited, all our other models use the standard ReLU.