Adaptive Input Representations for Neural Language Modeling
Table 5: Perplexity on wikitext-103 with different context sizes during training and inference. Training block size is the number of consecutive tokens considered during training. Inference context is the number of tokens provided at evaluation before scoring tokens.
['Train block size', 'Inference context size', 'Validation perplexity', 'Test perplexity']
Training block size is the number of consecutive tokens that are considered at once during training. Inference context is the number of tokens that are provided at evaluation before any tokens are scored. Simply increasing the training block size from 512 to 3072 results in an improvement of nearly 1.2 perplexity with no inference context window. Increasing the context size at inference time results in an improvement of 0.6 perplexity for the largest training block size.