On the use of BERT for Neural Machine Translation
Table 6: IWSLT dataset results. IWSLT.FT: encoder is initialised with BERT model trained on IWSLT data; tbase/tbig: transformer base/big architecture for NMT model; dec3: decoder layers reduced for 6 to 3; bpe10k/bpe32k : amount of BPE merge operations used for source language, learnt on the same dataset as BERT model (IWSLT or Wiki+News).
['[EMPTY]', 'en-de', 'en-ru']
First, we observe that BERT pretrained model improves over the baseline, in any settings (BPE vocabulary, model architecture, dataset used for pretraining). In particular, it is interesting to mention that without pretraining, both tbig.bpe32k and tbig.bpe10k models diverge when trained on IWSLT. However, BERT pretraining gives a better initialization point, and allows to achieve very good performance both for en-de and en-ru. Thus, such pretraining can be an interesting technique in low-resource scenarios.