Finding Better Subword Segmentation for Neural Machine Translation
Table 2: BLEU scores with frequency-weighted schemes on German-English test set
['[ITALIC]  [BOLD] N', '[BOLD] FRQ′-BPE  [BOLD] Greedy', '[BOLD] FRQ′-BPE  [BOLD] Beam', '[BOLD] AV′-BPE  [BOLD] Greedy', '[BOLD] AV′-BPE  [BOLD] Beam', '[BOLD] DLG′-BPE  [BOLD] Greedy', '[BOLD] DLG′-BPE  [BOLD] Beam']
For greedy search, both DLG′-BPE and AV′-BPE achieve better performance than FRQ′-BPE and the best score is obtained with DLG′-BPE of 20K merge operations. For beam search, AV′-BPE and DLG′-BPE outperform FRQ-BPE′ with improvement of 0.47 and 0.37 BLEU points when N=20K respectively. In general, AV′-BPE and DLG′-BPE are comparable with each other and both superior to FRQ′-BPE with a margin.