What do Neural Machine Translation Models Learn about Morphology?
Table 3: POS tagging accuracy using encoder and decoder representations with/without attention.
['Attn', 'POS Accuracy ENC', 'POS Accuracy DEC', 'BLEU Ar-En', 'BLEU En-Ar']
There is a modest drop in representation quality with the decoder. This drop may be correlated with lower BLEU scores when translating English to Arabic vs. Arabic to English. To test this hypothesis, we train NMT models with and without attention and compare the quality of their learned representations. It seems that the decoder does not rely on the attention mechanism to obtain good target word representations, contrary to our hypothesis.