Jointly Learning to Align and Translate with Transformer Models
Table 2: Aer [%] per layer for all three language pairs: German→English (DeEn), Romanian→English (RoEn) and English→French (EnFr).
['Layer', 'DeEn', 'RoEn', 'EnFr']
This indicates that the initial layers are focusing more on learning good representations of the sentence generated by the decoder so far by self attention. Once good contextual representations are learnt, the higher layers fetch the relevant representations from the encoder ’s output via the encoder-decoder attention. However, interestingly the penultimate layer outperforms the final layer suggesting the final layer uses the alignment-based features in the penultimate layer to derive its own representation.