Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
Table 3: (a) Results for word-based, BPE-based and char-based models. (b) Results for different existing NMT techniques.
['Word, BPE and Char Word', 'Word, BPE and Char 29.4']
The answer is not immediately clear because, on the one hand, the data sparsity issue with word-based models is less severe when massive training data is available, so word-based models might be useful. Using the 4B subset on which the best performance is achieved in the topic-data-split setting, we examine the performances of the word-based model (vocabulary set to 50K), the subword BPE model (vocabulary set to 50K) and the char-based model (vocabulary set to 10K). The three models only differ in the encoding stage. Combined with fine-tuning, the three models achieve BLEU scores of 29.4, 29.8 and 30.1, respectively. This confirms the finding in Meng et al. As can be seen, monolingual ML fusion and back-translation actually harm the performance. This is expected since the monolingual dataset is significantly smaller than the pretraining bilingual datasets. Agreement reranking introduces a +0.20 BLEU boost, confirming the importance of the order by which sequences are generated. The improvement from diverse decoding is also tiny. We think that this might because the model is already good enough for using beam search to find the global optimal. Another significant improvement comes from the RL strategy, leading to a roughly +0.2 BLEU boost. The combination of RL, agreement-ranking and diverse decoding pushes the performance up to 32.3.