Adaptive Input Representations for Neural Language Modeling
Table 6: Perplexity on wikitext-103 when regularizing rare words in adaptive softmax.
['[EMPTY]', 'Tail dropout', 'Validation perplexity']
We also found that adaptive softmax can benefit from additional regularization of rare words. Adaptive softmax first projects the model output to the dimension of a particular cluster and then computes a dot product with the respective word embeddings. We add dropout to the output of the first projection for all clusters, except for the head.