Machine Translation Evaluation with Neural Networks
Table 3: Average Kendall’s τ on WMT12 for semantic vectors trained on different text collections. Shown are results (i) when using the semantic vectors alone, and (ii) when combining them with 4metrics and syntax25. The improvements over Wiki-GW50 are marked in bold.
['[BOLD] Source', '[BOLD] Alone', '[BOLD] Comb.']
One interesting aspect to explore is the effect of the dimensionality of the input embeddings. Here, we studied the impact of using semantic vectors of bigger sizes, trained on different and larger text collections. We can see that, compared to the 50-dimensional Wiki-GW50, 300-400 dimensional vectors are generally better by 1-2 τ points absolute when used in isolation; however, when used in combination with 4metrics+syntax25, they do not offer much gain (up to +0.2), and in some cases, we observe a slight drop in performance. We suspect that the variability across the different collections is due to a domain mismatch. Yet, we defer this question for future work.