A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions
Table 5: Average Kendall’s tau distance between the word alignments obtained after translating the test set with each MT system being evaluated and a monotone alignment (left); and average Kendall’s tau distance between the word alignments obtained for each MT system’s translation and the word alignments of the reference translation (right). Larger values represent more similar alignments. If the difference between the distances depicted in the two last columns is statistically significant according to paired bootstrap resampling [Koehn2004] with p=0.05 and 1000 iterations, the largest distance is shown in bold.
['[BOLD] Language direction', '[BOLD] Monotone vs.  [BOLD] PBMT', '[BOLD] Monotone vs.  [BOLD] NMT', '[BOLD] Monotone vs.  [BOLD] Ref.', '[BOLD] PBMT vs. Ref.', '[BOLD] NMT vs. Ref.']
The amount of reordering performed by each system can be estimated as the distance between the word alignments produced by that system and a monotone word alignment. The similarity between the reorderings produced by each MT system and the reorderings in the reference translation can also be estimated as the distance between the corresponding word alignments. The average over all the sentences in the test set of the distance proposed by Birch \shortcitebirch2011reordering is depicted.