Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis
Table 1: Perplexity of trained models by number of training sentences. All but the language models are 1000D BiLSTMs (500D per direction). The 500D forward and backward language models are combined into a single bidirectional language model for analysis experiments.
['[BOLD] Task', '[BOLD] Layer Size', '[BOLD] Attn.', '[BOLD] 1 Million', '[BOLD] 5 Million', '[BOLD] 15 Million', '[BOLD] 63 Million']
We use the same training procedure for all our models. We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets, and evaluate approximately every 5 million sentences when training on the larger datasets. We use SGD with an initial learning rate of 1. Whenever a model’s validation loss increases relative to the previous evaluation, we halve the learning rate and stop training when the learning rate reaches 0.515. For each training task and dataset size, we select the model with the lowest validation perplexity to perform auxiliary task evaluations on. For translation we use beam search (B=5) when decoding.