Language Modeling with Gated Convolutional Networks
Table 3: Results for single models on the WikiText-103 dataset.
['Model', 'Test PPL', 'Hardware']
On Google Billion Word, the average sentence length is quite short — only 20 words. We evaluate on WikiText-103 to determine if the model can perform well on a dataset where much larger contexts are available. On WikiText-103, an input sequence is an entire Wikipedia article instead of an individual sentence - increasing the average length to 4000 words. However, the GCNN outperforms LSTMs on this problem as well The GCNN-8 model has 8 layers with 800 units each and the LSTM has 1024 units. These results show that GCNNs can model enough context to achieve strong results.