Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
Table 4: sacreBLEU for shortcut variants of EN→DE models; ‘test mean’ averages over test-sets in table (1).
['Model', 'newstest 2017', 'newstest 2018', 'test mean']
To investigate whether the proposed method can confer a similar benefit to the transformer, we apply lexical shortcuts to decoder-to-encoder attention, replacing or adding to shortcuts feeding into self-attention. Formally, this equates to fixing E to Eenc in Eqn. Interestingly, combining both methods yields worse translation quality than either one does in isolation, indicating that the decoder is unable to effectively consolidate information from both source and target embeddings, which negatively impacts its learned latent representations. We therefore conclude that lexical shortcuts are most beneficial to self-attention. While adding shortcuts improves translation quality, it is not obvious whether this is predominantly due to improved accessibility of lexical content, rather than increased connectivity between network layers, as suggested in Dou et al. To isolate the importance of lexical information, we equip the transformer with non-lexical shortcuts connecting each layer n to layer n−2 , e.g. layer 6 to layer 4. As a result, the number of added connections and parameters is kept identical to lexical shortcuts, whereas lexical accessibility is disabled, allowing for minimal comparison between the two configurations. suggests that while non-lexical shortcuts improve over the baseline model, they perform noticeably worse than lexical connections. Therefore, the increase in translation quality associated with lexical shortcuts is not solely attributable to a better signal flow or the increased number of trainable parameters.