Analyzing Word Translation of Transformer Layers
Table 2: Translation Performance of Encoder Layers.
['Layer 0', 'BLEU1 33.1', 'Δ', 'BLEU 7.92', 'Δ']
To achieve this goal, we feed output representations from an encoder layer to the corresponding linear projection layer, and feed the output of the linear projection layer directly to the decoder classifier, and retrieve tokens with highest probabilities as “translations”. Even though such “translations” from encoder layers have a same length and a same word-order as source sentences, individual source tokens are translated to the target language to some extent. “FULL” is the performance of the whole Transformer model (decoding with a beam size of 4). “Δ” means the improvements obtained by the introduced layer (or the decoder for “FULL”) over the previous layer. It is reasonable that encoder layers achieve a comparably high BLEU 1 score while a low BLEU score, as they perform word translation in the same order as the source sentence without any word re-ordering of the target language. We suggest the BLEU 1 score achieved by only the source embedding layer (i.e. translating with only embeddings) surprising and worth noting.