“Bilingual Expert” Can Find Translation Errors
Table 1: Results of sentence level QE on WMT 2017. MD: model derived features. MM: mis-matching features.
['Method', 'test 2017 en-de  [BOLD] Pearson’s ↑', 'test 2017 en-de MAE ↓', 'test 2017 en-de RMSE ↓', 'test 2017 en-de  [BOLD] Spearman’s ↑', 'test 2017 en-de DeltaAvg ↑', 'test 2017 de-en  [BOLD] Pearson’s ↑', 'test 2017 de-en MAE ↓', 'test 2017 de-en RMSE ↓', 'test 2017 de-en  [BOLD] Spearman’s ↑', 'test 2017 de-en DeltaAvg ↑']
Unbabel is combination of a feature-rich sequential linear model with a neural network. POSTECH is a predictor-estimator model with all Bi-GRU modules. Baseline is the official provided system. The primary metrics of sentence level task are Pearson’s correlation and Spearman’s rank correlation of the entire testing data. Alternatively, mean average error (MAE), root mean squared error (RMSE), or the average of delta values (DeltaAvg) can also measure the performance of overall predictions, but not be a ranking reference in the QE task. For both single and ensemble model comparisons, our algorithm can outperform all other systems for the two primary metrics. The ranking results are generated by the predicted HTER scores. In addition, we also analyze the importance of model derived features (MD) and mis-matching features (MM) the ablation study. With 4-dimensional mis-matching features alone, the model can still achieve comparable or better performance than the second single system last year. It demonstrates that the low dimensional features can provide a strong prediction signal as well.