Rational Recurrences
Table 3: Language modeling perplexity on PTB test set (lower is better). LSTM numbers are taken from Lei et al. (2017b). ℓ denotes the number of layers. Bold font indicates best performance.
['[BOLD] Model', 'ℓ', '[BOLD] # Params.', '[BOLD] Dev.', '[BOLD] Test']
Following Collins et al. The middle block compares all models with two layers and 10M trainable parameters. rrnn(B) and rrnn(C) achieve roughly the same performance; interpolating both unigram and bigram features, rrnn(F) outperforms others by more than 2.9 test perplexity. For the three-layer and 24M setting (the bottom block), we observe similar trends, except that rrnn(C) slightly underperforms rrnn(B). Here rrnn(F) outperforms others by more than 2.1 perplexity.