Regressing Word and Sentence Embeddingsfor Regularization of Neural Machine Translation
TABLE III: BLEU scores over the Eu-En test set. The reported results are the average of 5 independent runs.
['[BOLD] Models', 'Word/BPE', 'BLEU']
Moreover, ReWE+ReSE has been able to improve the results even further (+3.15 BLEU pp when using BPE and +5.15 BLEU pp at word level over the corresponding baselines). Basque is, too, a morphologically-rich language and using BPE has proved very beneficial (+4.27 BLEU pp over the best word-level model). As noted before, the eu-en dataset is very low-resource (less than 100,000 sentence pairs) and it is more likely that the baseline models generalize poorly. Consequently, regularizers such as ReWE and ReSE are more helpful, with larger margins of improvement with respect to the baselines. On a separate note, the transformer has unexpectedly performed well below the LSTM on this dataset, and especially so with BPE. We speculate that it may be more sensitive than the LSTM to the dataset’s much smaller size, or in need of more refined hyper-parameter tuning.