Adaptive Input Representations for Neural Language Modeling
Table 4: Test perplexity on billion word. Training speed measured on four 8-GPU machines.
['[EMPTY]', '[BOLD] Input', '[BOLD] Output', '[BOLD] Valid', '[BOLD] Test', '[BOLD]  Train time (hours) ', '[BOLD] Params']
ADP-T is 34% faster than ADP because there are fewer parameters to update. Similar to before, ADP-T trains more than twice as fast as CNN at higher accuracy , however, the accuracy gap is narrower than for wikitext-103. Regularization is more important on wikitext-103 while models for billion word benefit from additional capacity. Because of this we used input word embeddings of size 256 for ASM.