Regressing Word and Sentence Embeddingsfor Regularization of Neural Machine Translation
TABLE IV: BLEU scores over the De-En test set. The reported results are the average of 5 independent runs.
['[BOLD] Models', 'Word/BPE', 'BLEU']
For this dataset, we have only been able to perform experiments with the LSTM due to the exceedingly long training times of the transformer. At word level, both ReWE and ReWE+ReSE have been able to outperform the baseline, although the margins of improvement have been smaller than for the other language pairs (+0.42 and +0.48 BLEU pp, respectively). However, when using BPE both ReWE and ReWE+ReSE have performed slightly below the baseline ( −0.37 and −0.12 points BLEU pp, respectively). This shows that when the training data are abundant, ReWE or ReSE may not be beneficial. To probe this further, we have repeated these experiments by training the models over subsets of the training set of increasing size (200K, 500K, 1M, and 2M sentence pairs). Fig. The plot clearly shows that the performance margin increases as the training data size decreases, as expected from a regularized model.