Pairwise Neural Machine Translation Evaluation
Table 1: Kendall’s tau (\tau) on the WMT12 dataset for various metrics. Notes: (i) the version of Meteor that took part in the WMT12 competition (marked with {}^{*} in section IV of the table) is different from the one used in our experiments (section I of the table), (ii) values marked as na were not reported by the authors.
['[BOLD] I', '[BOLD] System 4metrics [BOLD] : commonly-used individual metrics', '[BOLD] Details 4metrics [BOLD] : commonly-used individual metrics', '[BOLD] Kendall’s \\tau  [BOLD] cz', '[BOLD] Kendall’s \\tau  [BOLD] de', '[BOLD] Kendall’s \\tau  [BOLD] es', '[BOLD] Kendall’s \\tau  [BOLD] fr', '[BOLD] Kendall’s \\tau  [BOLD] AVG']
We will refer to the set of these four metrics as 4metrics. These metrics are not tuned and achieve Kendall’s \tau between 18.5 and 23.5. and Wiki-GW25. These networks achieve modest \tau values around 10, which should not be surprising: they use very general vector representations and have no access to word or n-gram overlap or to length information, which are very important features to compute similarity against the reference. However, as will be discussed below, their contribution is complementary to the four previous evaluation metrics and will lead to significant improvements in combination with them. We can see that just combining the four metrics in a flat neural net (i.e., no hidden layer), which is equivalent to a logistic regression, yields a \tau of 27.06, which is better than the best of the four metrics by 3.5 points absolute, and also better by over 1.5 points absolute than the best metric that participated at the WMT12 metrics task competition (spede07pP with \tau=25.4). Indeed, 4metrics is a strong mix that involves not only simple lexical overlap but also approximate matching, paraphrases, edit distance, lengths, etc. Yet, adding to 4metrics the embedding vectors yields sizeable further improvements: +1.5 and +2.0 points absolute when adding syntax25 and Wiki-GW25, respectively. Finally, adding both yields even further improvements close to \tau of 30 (+2.64 \tau points), showing that lexical semantics and syntactic representations are complementary.