Pyramidal Recurrent Unit for Language Modeling
Table 4: Impact of different sub-sampling methods on the word-level perplexity (lower is better). We used g=1 and K=4 in our experiments.
['[BOLD] Dataset', '[BOLD] Skip', '[BOLD] Max pool', '[BOLD] Avg. Pool', '[BOLD] Convolution']
Average pooling performs the best while skipping give comparable performance. Both of these methods enable the network to learn richer word representations while representing the input vector in different forms, thus delivering higher performance. Surprisingly, a convolution-based sub-sampling method does not perform as well as the averaging method. The tanh function used after convolution limits the range of output values which are further limited by the LSTM gating structure, thereby impeding in the flow of information inside the cell. Max pooling forces the network to learn representations from high magnitude elements, thus distinguishing features between elements vanishes, resulting in poor performance.