Long Distance Relationships without Time Travel: Boosting the Performance of a Sparse Predictive Autoencoder in Sequence Modeling
Table 2. Language modeling results. bRSM variants with each of 4 added feature ablated are shown. †: As reported by Mikolov et al (mikolovEmpiricalEvaluationCombination).
['Model', 'Test PPL', 'No. of params']
The previous state of the art was an LSTM with the addition of mutual gating of the current input and the previous output reporting 44.8 PPL (melisMogrifierLSTM2019). bRSM, with and without the word cache, outperforms all early language modeling architectures, including ordinary (non-gated) recurrent neural language models trained with BPTT. While these results are not yet competitive with state-of-the-art deep models such as the Transformer, and modern LSTM-based approaches, they demonstrate a significant step forward for resource efficient performance.