Language Modeling with Gated Convolutional Networks
Table 2: Results on the Google Billion Word test set. The GCNN outperforms the LSTMs with the same output approximation.
['Model', 'Test PPL', 'Hardware']
We find the GCNN outperforms the comparable LSTM results on Google billion words. To accurately compare these approaches, we control for the same number of GPUs and the adaptive softmax output model (Grave et al., We evaluate the throughput and responsiveness for models that reach approximately 43.9 perplexity on the Google Billion Word benchmark. A bottleneck block wedges a k>1 convolution between two k=1 layers. This designs reduces computational cost by reducing and increasing dimensionality with the k=1 layers so that the convolution operates in a lower dimensional space. Our results show that the use of bottleneck blocks is important to maintaining computational efficiency.