Quasi-Recurrent Neural Networks
Table 2: Single model perplexity on validation and test sets for the Penn Treebank language modeling task. Lower is better. “Medium” refers to a two-layer network with 640 or 650 hidden units per layer. All QRNN models include dropout of 0.5 on embeddings and between layers. MC refers to Monte Carlo dropout averaging at test time.
['[BOLD] Model', '[BOLD] Parameters', '[BOLD] Validation', '[BOLD] Test']
The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of Zaremba et al. This may be due to the limited computational capacity that the QRNN’s pooling layer has relative to the LSTM’s recurrent weights, providing structural regularization over the recurrence.