Character-based NMT with Transformer
Table 2: Similarity metrics between test sets and training sets.
['Dataset', '# Sents', 'DE-EN % Unseen', 'DE-EN PPL', 'EN-DE % Unseen', 'EN-DE PPL', 'DE-EN (high) % Unseen', 'DE-EN (high) PPL']
“% Unseen” is the percentage of words in the test set that are not present in the training corpus. “PPL” is the perplexity measure of the test set using a language model trained on the training data. DE-EN low resource. Character level models are better for all out of domain datasets, except for Europarl. This suggests that character level models outperform BPE when evaluated on data sufficiently different from the training domain in this low resource setting. DE-EN high resource. Character level models are now only better when testing on the WMT-Biomedical test set. For all other test sets, BPE 30 000 leads to the best BLEU scores. EN-DE low resource. We see similar performance for in and out of domain data. Good BPE models on in-domain test set are still better on out-of-domain test sets. A possible explanation is the lower proportion of unseen words as compared to German and seeing the words more frequently in the training corpus.