Neural Machine Translation with Byte-Level Subwords
Table 4: Average lengths of training sentences tokenized with different vocabularies.
['[EMPTY]', '[EMPTY]', 'Byte 256', 'BBPE 1K', 'BBPE 2K', 'BBPE 3K', 'BBPE 4K', 'BBPE 8K', 'BBPE 11K', 'BBPE 16K', 'BBPE 32K', 'Char 3K', 'Char 8K', 'Char 11K', 'BPE 8K', 'BPE 16K', 'BPE 32K']
Sx4SSx1SSS0Px3 Impact on Sequence Lengths Compared to BPE, BBPE symbols are generally finer-grained with shorter byte-level lengths, which results in longer tokenized sequences as well as longer training and inference time. BBPE, however, is optimized for compression-based objective (the same as BPE), and is still more efficient than character vocabulary. We can observe that sentences tokenized with BBPE have significantly shorter lengths than the character ones, even when the BBPE vocabulary is much smaller (for example only 1/5 of character set size on X-En). Another observation is that source-target length ratio for BBPE tends to be much larger when source character set and target character set have very different sizes (for example 11K for X-En source side and 0.1K for the target side). And this situation becomes more severe when BBPE vocabulary size increases. In this case, alignments may be more difficult to learn during model training, since target tokens need attentions on multiple source tokens more often.