Learning variable length units for SMT between related languages via Byte Pair Encoding
Table 7: Pearson’s correlation coefficient between lexical similarity and translation accuracy (both in terms of LCSR at character level). This was computed over the test set between: (i) sentence level lexical similarity between source and target sentences and (ii) sentence level translation accuracy between hypothesis and reference.
['[BOLD] Src-Tgt', '[BOLD] Word', '[BOLD] Morph', '[BOLD] BPE', '[BOLD] OS', '[BOLD] Char']
We see that character models work best when the source and target sentences are lexically very similar. The additional context decouples OS and BPE units from lexical similarity. Words and morphemes show the least correlation since they do not depend on lexical similarity.