Language Modeling with Gated Convolutional Networks
Table 4: Processing speed in tokens/s at test time for an LSTM with 2048 units and GCNNs achieving 43.9 perplexity on Google Billion Word. The GCNN with bottlenecks improves the responsiveness by 20 times while maintaining high throughput.
['[EMPTY]', 'Throughput (CPU)', 'Throughput (GPU)', 'Responsiveness (GPU)']
The throughput of the LSTM is measured by using a large batch of 750 sequences of length 20, resulting in 15,000 tokens per batch. The responsiveness is the average speed to process a sequence of 15,000 contiguous tokens. The LSTM performs very well on GPU because the large batch size of 750 enables high parallelization over different sentences. This is because the LSTM implementation has been thoroughly optimized and uses cuDNN, whereas the cuDNN implementation of convolutions is not been optimized for the 1-D convolutions we use in our model. We believe much better performance can be achieved by a more efficient 1-D cuDNN convolution. Unlike the LSTM, the GCNN can be parallelized both over sequences as well as across the tokens of each sequence, allowing the GCNN to have 20x higher responsiveness.