Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
Table 3: Sentence length of different translation granularities.
['Language', 'word', 'character', 'Hybrid', 'BPE', 'WPM', 'WPM(raw)']
We execute different translation granularities on the training corpus. To make a comparison, We randomly choose 10000 sentences. A well-known flaw of NMT model is the inability to properly translate long sentences. However, most of translation granularities will go below the word level. We wonder what the translation performance of different lengths are on all translation granularities. We follow Bahdanau et al.