Pyramidal Recurrent Unit for Language Modeling
Table 1: Comparison of single model word-level perplexity of our model with state-of-the-art on validation and test sets of Penn Treebank and Wikitext-2 dataset. For evaluation, we select the model with minimum validation loss. Lower perplexity value represents better performance.
['[BOLD] Model', '[BOLD] WT-2  [BOLD] Params', '[BOLD] WT-2  [BOLD] Val', '[BOLD] WT-2  [BOLD] Test', '[BOLD] PTB  [BOLD] Params', '[BOLD] PTB  [BOLD] Val', '[BOLD] PTB  [BOLD] Test']
Language modeling is a fundamental task for NLP and has garnered significant attention in recent years (see Merity et al. Yang et al. with the mixture of softmaxes (MoS) technique, which increases the rank of the matrix used to compute next-token probabilities. Further, Merity et al. Our work is complementary to these recent softmax layer and inference procedure improvements. We can see that the PRU achieves the best performance with fewer parameters. Fixed K and variable g: When we vary the value of g at fixed number of pyramidal levels K, the total number of recurrent unit parameters decreases significantly with a minimal impact on the perplexity. For example, PRUs with K=2 and g=4 learns 77% fewer recurrent unit parameters while its perplexity (lower is better) increases by about 12% in comparison to LSTMs.