Machine Translation Evaluation with Neural Networks
Table 6: Kendall’s τ on WMT12 for neural networks using different variants of word-embedding fine tuning. All variants are implemented on top of our full system from Section 4, referred to as “Pre-computed sentence embeddings” in this table.
['[BOLD] Details', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] cz ', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] de', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] es', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] fr', '[BOLD] Kendall’s  [ITALIC] τ  [BOLD] AVG']
, we observe the results on the WMT12 dataset measured by Kendall’s τ in the different settings described above. First, note that the effect of normalization is noticeable. Just by switching to a dynamic composition of sentence-level embeddings, we lose 0.16 points absolute. Allowing the moderate fine-tuning of the embedding matrix only improves performance slightly by 0.07. However, allowing the full fine-tuning of the embedding matrix, yields an improvement of 0.25 over the un-tuned setting, and even slightly over the fully normalized baseline system (+0.09).