Approximating Stacked and Bidirectional Recurrent Architectures with the Delayed Recurrent Neural Network
Table 3: Performance for smaller networks on the masked character-level language modeling task. Mean and standard deviation values are computed over 5 repetitions of training and inference runtime on the test set.
['Model', 'Layers', 'Delay', 'Units', 'Params.', 'Val. BPC', 'Test BPC', 'Runtime']
As expected, the stacked Bi-LSTMs achieve the lowest BPC. However, as the number of layers increases, the inference runtime also increases because of the synchronization needed between layers. Notably, d-LSTMs with intermediate delays achieve a BPC that is within 5% of the Bi-LSTM with at least 4× faster runtime. Since all of the d-LSTMs have a single layer, inference runtime remains constant as the delay and the capacity of these networks increases. We find similar results for other network capacities (see supplementary material). We sampled more delay values for d-LSTMs, but the general conclusions remain the same: intermediate values of delay achieve the lowest BPC. Forward-pass runtimes across delay values show a small increase with larger delays, but the increment is relatively flat compared to stacked LSTMs or (stacked) Bi-LSTMs as they increase in depth. For these experiments, we also used a batch of 128 sequences, and an embedding of dimension 10.