Adaptive Input Representations for Neural Language Modeling
Table 1: Test perplexity on billion word. Adaptive inputs share parameters with an adaptive softmax. Training times of Char-CNN and Adaptive input models are measured when training with 64 GPUs. †does not include embedding and softmax layers
['[EMPTY]', '[BOLD] Test', '[BOLD]  Train Time (hours) ', '[BOLD] Parameters']
Test perplexity on wikitext-103 (cf. Training time is based on 8 GPUs. The adaptive input model outperforms the best previously reported result at an order of magnitude fewer parameters. Our large model performs nearly as well as an ensemble of over ten models and achieves a new state of the art of 24.14 perplexity. Our very large model performs as well as an ensemble of over ten models and achieves 23.02 perplexity. The Char-CNN model performs 0.6 PPL worse than the standard adaptive input model even though it trained for over 40% longer.