What do Neural Machine Translation Models Learn about Morphology?
Table 4: POS tagging accuracy using word-based and char-based encoder/decoder representations.
['[EMPTY]', 'POS Accuracy ENC', 'POS Accuracy DEC', 'BLEU Ar-En', 'BLEU En-Ar']
In both bases, char-based representations perform better. BLEU scores behave differently: the char-based model leads to better translations in Arabic-to-English, but not in English-to-Arabic. A possible explanation for this phenomenon is that the decoder’s predictions are still done at word level even with the char-based model (which encodes the target input but not the output). In practice, this can lead to generating unknown words. Indeed, in Arabic-to-English the char-based model reduces the number of generated unknown words in the MT test set by 25%, while in English-to-Arabic the number of unknown words remains roughly the same between word-based and char-based models.