Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
Table 3: sacreBLEU scores for small EN→DE models; ‘test mean’ denotes the average of test-sets in table (1).
['Model', 'newstest 2017', 'newstest 2018', 'test mean']
Next, we investigate the interaction between the number of model parameters and improvements in translation quality afforded by the proposed lexical connections. the model’s capacity is sufficiently large. To establish whether this decline in effectiveness is gradual, we scale down the standard transformer, halving the size of its embeddings, hidden states, and feed-forward sub-layers. One explanation is that given sufficient capacity, the model is capable of accommodating the upward propagation of lexical features without having to neglect other sources of information. However, as long as the model’s representational capacity is within certain limits, the effect of lexical shortcuts remains comparable across a range of model sizes. With this in mind, the exact interaction between model scale and the types of information encoded in its hidden states remains to be fully explored. We leave a more fine-grained examination of this relationship to future research.