Word-Alignment-Based Segment-Level Machine Translation Evaluation using Word Embeddings
Table 2: Kendall’s τ correlations of word-alignment-based methods and the official human judgements for each dataset. (WMT12, WMT13, and WMT15: European–English datasets, and WAT2015 and NTCIR8: Japanese–English datasets)
['Evaluation Metrics', 'WMT12', 'WMT13', 'WMT15', 'WAT2015', 'NTCIR8']
MAS has the highest correlation score for the WMT datasets, whereas AAS has the highest correlation score for WAT2015 and NTCIR8 datasets. In particular, Kendall’s τ score of HAS in WMT12 exhibits very low correlation. It seems that the 1:1 alignment is too strict to calculate sentence similarity in MT evaluation, while the 1:m (MAS) alignment performs well, possibly because of the removal of noisy word alignment. On the other hand, AAS is more stable than MAS and HAS for WAT2015 and NTCIR8 datasets. As a rule of thumb, AAS with high threshold values (0.6–0.9) shows stable high correlation across all language pairs, but if it is possible to use development data to tune the parameters, MAS with different values of thresholds should be considered.