On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
Table 1: Pearson correlations with segment-level human judgments on the WMT17 dataset.
['Setting', 'Metrics', 'cs-en', 'de-en', 'fi-en', 'lv-en', 'ru-en', 'tr-en', 'zh-en', 'Average']
We note that re-mapping has considerable effect for M-BERT (up to 10 points improvements), but much less so for LASER. We believe that this is because the underlying embedding space of LASER is less ‘misaligned’ since it has been (pre-)trained on parallel data. While the re-mapping is thus effective for metrics based on M-BERT, we still require the target-side LM to outperform BLEU.