Variable Computation in Recurrent Neural Networks
Table 3: Left: Bits per character for character level language modeling on Penn TreeBank. CRNN refers to the Conditional RNN from (Bojanowski et al., 2015). Middle: Bits per bit for bit level language modeling on Penn TreeBank. Right: Bits per character for character level language modeling on Text8. ∗From (Zhang et al., 2016)
['[BOLD] Bit level PTB unit type', '[BOLD] Bit level PTB RNN-d', '[BOLD] Bit level PTB bpb']
In each of our experiments, we are interested in investigating two specific aspects of our model. On the one hand, do the time patterns that emerge agree with our intuition of the time dynamics expressed in the data? On the other hand, does the Variable Computation Unit (VCU) yield a good predictive model? More specifically, does it lead to lower perplexity than a constant computation counterpart which performs as many or more operations? Note that the computational complexity gains we refer to are exclusively in terms of lowering the number of operations, which does not necessarily correlate with a speed up of training when using general purpose GPU kernels; it is however a prerequisite to achieving such a speed up with the proper implementation, motivating our effort.