On the Importance of Word Boundaries in Character-level Neural Machine Translation
Table 2: Results of the evaluation of models in capturing morphological variations in the output using the Morpheval English-German test set. The accuracy is measured with the percentage of correctly captured morphological contrasts. The best scores for each translation direction are in bold font.
['[BOLD] Variation', '[BOLD] Chars', '[BOLD] Subwords', '[BOLD] Hier.']
Since solely relying on BLEU scores may not be sufficient in understanding the generative properties of different NMT models, we perform an additional evaluation in order to assess the capacity of models in learning syntactic or morphological dependencies using the Morpheval test suites, which consist of sentence pairs that differ by one morphological contrast, and each output accuracy is measured in terms of the percentage of translations that could convey the morphological contrast in the target language. The results of our analysis support the benefit of using BPE in German as an open-vocabulary NMT solution, where the subword-based model obtains the highest accuracy in most of the morphological paradigm generation tasks. The character-level model shows to be promising in capturing few morphological features better than the former, such as negation or comparative adjectives, and in capturing agreement features, the hierarchical decoding model generally performs better than the subword-based model. The dominance of subword-based models could be due to the high level of compounding in German where segmentation is possibly beneficial in splitting compound words and aiding better syntactic modeling in some cases. These results generally suggest the importance of processing the sentence context at the word level in order to induce a better notion of syntax during generation.