Language Modeling with Deep Transformers
Table 9: Effect of sinusoidal positional encoding. Perplexity after 5 epochs (13M updates) for (L, 2048, 512, 8) models.
['[ITALIC] L', 'Position. encoding', 'Params. in M.', 'Perplexity Train', 'Perplexity Dev', 'Perplexity Test']
In the autoregressive problem where a new token is provided to the model at each time step, the amount of information the model has access to strictly increases from left to right at the lowest level of the network; the deeper layers should be able to recognize this structure which should provide the model with some positional information by its own. To check this hypothesis, we train models without any positional encoding.