The Missing Ingredient in Zero-Shot Neural Machine Translation
Table 3: BLEU on subset of examples predicted in the right language by zero-shot translation through the multilingual model (newstest2012)
['[EMPTY]', '# examples', 'Pivot', 'Zero-Shot']
Given that a large portion of the errors are due to incorrect language, we try to estimate the improvement to zero-shot translation quality that could potentially be achieved by solving this issue. We discount these errors by re-evaluating the BLEU scores of zero-shot translation and pivoting on only those examples that the multilingual model already zero-shot translates to the right language. We find that although the vanilla zero-shot translation system is much stronger than expected at first glance, it still lags the pivoting by 0.5 BLEU points on French to German and by 2.7 BLEU points on German to French. This gap and the below analysis indicate a generalization problem in the model.