Novel Applications of Factored Neural Machine Translation
Table 2: Simulated low-resource condition results (in %). All systems were trained on a corpus of 500K randomly sampled En→De sentences.
['[BOLD] English→German', 'Bleu  [BOLD] newstest 2017', 'Ter  [BOLD] newstest 2017', 'Bleu  [BOLD] newstest 2019', 'Ter  [BOLD] newstest 2019']
For comparison, we simulate low-resource conditions on the En→De task by randomly sampling a subset of 500K sentence pairs from the training corpus. To prevent overfitting, we used only 2 encoder layers and label smoothing of 0.2 for these experiments. In addition, we optimized the size of the subword vocabulary to 10K. Casing prediction via a factor improves over the baseline by 0.8% \textscBleu on newstest2019. Factorized subword merging harms translation quality on newstest2017. For OSNMT, we again see a weak performance of the baseline, but a big improvement due to factoring out SRC_POP tokens. The lower-cased vocabulary of size 50K for the casing prediction experiments was able to represent a total of 134K distinct word casing variants from the training data. In low-resource settings, the 10K vocabulary represented 23K casing variants. We suspect that with enough training examples the baseline model is able to learn connections between different casing variants of a word, therefore a separate casing prediction does not improve translation quality. In low-resource settings, however, explicitly sharing information between casing variants can mitigate the data sparseness problem.