On the Linguistic Representational Power of Neural Machine Translation Models
Table 13: BLEU scores and morphological classifier accuracy across language pairs, comparing (encoders of) fully character-based LSTM and charCNN LSTM models.
['[EMPTY]', 'BLEU de-en', 'BLEU cs-en', 'Accuracy de-en', 'Accuracy cs-en']
The character-based representations consistently outperformed other representations on all language pairs, while the word-based representations achieved the lowest accuracy. The differences are more significant in the case of languages with relatively complex morphology, notably Czech and Russian. We see a difference of up to 14% in favor of using character-based representations when compared with the word-based representations. The improvement is minimal in the case of English (1.2%), which is a morphologically-simpler language. Comparing subword units as obtained using Morfessor and BPE, we found Morfessor to give much better morphological tagging performance, especially in the case of the morphologically-richer languages, Czech and Russian. The representations learned from morpheme-segmented units were found helpful in learning language morphology. The character-based segmentation gave higher BLEU scores compared to a BPE-based system in the case of the morphologically-rich language Czech, but character-based models performed poorly in the case of German, which requires handling long-distance dependencies. The character-based models reported in this paper were trained using bidirectional LSTM models only. We simply segmented words into characters and marked word boundaries. However, we did try charCNN kim2015character; costajussa-fonollosa:2016: P16-2 models in our preliminary experiments. The model is a CNN with a highway network over characters and trains an LSTM on top of it. We therefore left it out and focused on char-based LSTM model..