Machine Translation Evaluation with Neural Networks
Table 1: Kendall’s tau (τ) on the WMT12 dataset for various metrics. ‘AVG’ is the average τ for the four language pairs. The best results are marked in boldface.
['[BOLD] System  [BOLD] I. 4metrics [BOLD] : commonly-used individual metrics', '[BOLD] Details  [BOLD] I. 4metrics [BOLD] : commonly-used individual metrics', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] cz', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] de', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] es', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] fr', '[BOLD] AVG']
We will refer to the set of these four metrics as 4metrics. These metrics are not tuned and achieve Kendall’s τ between 18.5 and 23.5. These are the metrics that are added as pairwise similarity features in our neural network approach (skip arcs). These networks achieve modest τ values around 10, which should not be surprising: they use very general vector representations and have no access to word or n-gram overlap or to length information, which are very important features to compute similarity against the reference. However, as will be discussed below, their contribution is complementary to the four previous evaluation metrics and will lead to significant improvements in combination with them. Indeed, 4metrics is a strong mix that involves not only simple lexical overlap but also approximate matching, paraphrases, edit distance, lengths, etc. Yet, adding to 4metrics the embedding vectors yields sizeable further improvements: +1.5 and +2.0 points absolute when adding syntax25 and Wiki-GW50, respectively. Finally, adding both yields even further improvements close to τ of 30 (+2.64 τ points), showing that lexical semantics and syntactic representations are complementary.