Unsupervised Quality Estimation for Neural Machine Translation
Table 2: Pearson (r) correlation between unsupervised QE indicators and human DA judgments. Results that are not significantly outperformed by any method are marked in bold; results that are not significantly outperformed by any other method from the same group are underlined.
['[EMPTY]', '[BOLD] Method', '[BOLD] Low-resource Si-En', '[BOLD] Low-resource Ne-En', '[BOLD] Mid-resource Et-En', '[BOLD] Mid-resource Ro-En', '[BOLD] High-resource En-De', '[BOLD] High-resource En-Zh']
Unsupervised QE indicators are grouped as follows: We use the Hotelling-Williams test to compute significance of the difference between dependent correlations Williams For each language pair, results that are not significantly outperformed by any method are marked in bold; results that are not significantly outperformed by any other method from the same group are underlined. The use of multi-head attention with multiple layers in Transformer may also negatively affect the results. As shown by Voita et al. Therefore, combining the information coming from different heads and layers in a simple way may not be an optimal solution. To test whether this is the case, we computed attention entropy and its correlation with DA for all possible combinations of heads and layers. Using the best head/layer combination for QE is limited by the fact that it requires validation on a dataset annotated with DA and thus is not fully unsupervised. This outcome opens an interesting direction for further experiments to automatically discover the best possible head/layer combination.