Machine Translation Evaluation Using Bi-directional Entailment
Table 1: Pearson’s System Correlation on WMT’14 data for baseline and participating metrics. Bold results with symbol † denotes highest correlation and symbol †† denotes second highest correlation for a language pair.
['[BOLD] Metric/Model', '[BOLD] cs-en', '[BOLD] de-en', '[BOLD] fr-en', '[BOLD] ru-en', '[BOLD] Average', '[BOLD] SpearmanAvg']
Our metric shows better correlation with the human score than all other evaluation metrics for all language pairs except for cs-en pair. Our metric has an average correlation of 0.957, which is an improvement from 0.871 of BLEU. Compared to LAYERED, which uses UNL to capture semantics, our metric gives better correlation in all language pairs. We performed statistical significance test to validate results from multiple runs and found that results are statistically significant under the one-tailed paired t-test at the 99% signiﬁcance level.