On the use of BERT for Neural Machine Translation
Table 5: Robustness tests. BLEU scores for clean and ’noisified’ (with different noise type) news14 testset.
['Models', 'news14', '+UNK.S', '+UNK.E', '+chswap', '+chrand', '+up']
As expected, we observe an important drop in BLEU scores due to the introduced noise. We observe that most pretrained BERT models have better BLEU scores compared to baseline for all type of noise (except NMTSrc.FT which suffers more from unknown token introduction in the end of the sentence compared to the Baseline). However, these results are not enough to conclude, whether higher BLEU scores of BERT-augmented models are due to better robustness, or simply because these models are slightly better than the baseline in general.