Finding Better Subword Segmentation for Neural Machine Translation
Table 3: BLEU scores on Chinese-English test set (beam size=10)
['[BOLD] Source', '[BOLD] Target  [BOLD] FRQ′-BPE', '[BOLD] Target  [BOLD] AV′-BPE', '[BOLD] Target  [BOLD] DLG′-BPE']
Here, the subword segmentation learning is performed separately on the source and target sides with 30K merge operations which is tuned the same way as German-English task. We also try different goodness measures on each side at the same time. It can be found that DLG′-BPE on the both sides significantly outperforms FRQ′-BPE with 0.65 BLEU points. It can be seen that DLG′-BPE does much better at the source segmentation for Chinese than the other two, while AV′-BPE and FRQ′-BPE are on a par with each other.