Pyramidal Recurrent Unit for Language Modeling
Table 3: Impact of different transformations used for processing input and context vectors (LT - linear transformation, PT - pyramidal transformation, and GLT - grouped linear transformation). Here, † represents that PT was used without residual connection, PPL represents word-level perplexity (lower is better), and the number of parameters are in million. We used K=g=4 in our experiments.
['[EMPTY]', '[BOLD] Transformations', '[BOLD] Transformations', '[BOLD] PTB  [BOLD] PPL', '[BOLD] PTB  [BOLD] # Params', '[BOLD] WT-2  [BOLD] PPL', '[BOLD] WT-2  [BOLD] # Params']
Transformations ht−1. We make following observations: (1) Using the pyramidal transformation for the input vectors improves the perplexity by about 1 point on both the PTB and WT-2 datasets while reducing the number of recurrent unit parameters by about 14% (see R1 and R4). We note that the performance of the PRU drops by up to 1 point when residual connections are not used (R4 and R6). (2) Using the grouped linear transformation for context vectors reduces the total number of recurrent unit parameters by about 75% while the performance drops by about 11% (see R3 and R4). When we use the pyramidal transformation instead of the linear transformation, the performance drops by up to 2% while there is no significant drop in the number of parameters (R4 and R5).