Neural Machine Translation with Byte-Level Subwords
Table 6: Ja-En test BLEU scores.
['# of train samples', '# of train samples', 'KFTT 440K', 'TED 223K', 'JESC 2.8M', 'All 3.5M']
Languages using logographic writing systems, such as Chinese and Japanese, can have over 50K characters, while only a small portion of them are frequently used. Our Ja-En dataset has a set of 7.9K characters, where 99.99% tokens in the training set are covered by the top 2.4K characters. With this observation, we experiment with BBPE 4K which is roughly 50% of the character set size. We find that BBPE is comparable to BPE and even outperforms BPE when using larger Tbig model (