DiscoTK: Using Discourse Structure for Machine Translation Evaluation
Table 1: Evaluation results on WMT12 and WMT13 datasets at segment and system level for the main combined DiscoTK measures proposed in this paper.
['[EMPTY]', '[EMPTY]', 'Segment Level WMT12', 'Segment Level WMT13', 'System Level WMT12', 'System Level WMT12', 'System Level WMT13', 'System Level WMT13']
We evaluated the performance of the metrics on the WMT12 and WMT13 datasets both at the segment and the system level, and we used WMT11 as an additional tuning dataset. We measured the performance of the metrics in terms of correlation with human judgements. At the segment level, we evaluated using Kendall’s Tau (τ), recalculated following the WMT14 official Kendall’s Tau implementation. At the system level, we used Spearman’s rank correlation (ρ) and Pearson’s correlation coefficient (r). In all cases, we averaged the results over all into-English language pairs. The symbol ‘\mathchar1343\relax’ represents the untuned versions of our metrics, i.e., applying a uniform linear combination of the individual metrics. First, DiscoTKparty is better than DiscoTKlight in all settings, indicating that the discourse-based metrics are very well complemented by the heterogeneous metric set from Asiya. DiscoTKlight achieves competitive scores at the system level (which would put the metric among the best participants in WMT12 and WMT13); however, as expected, it is not robust enough at the segment level. On the other hand, the tuned versions of DiscoTKparty are very competitive and improve over the already strong Asiya in each configuration both at the segment- and the system-level. The improvements are small but consistent, showing that using discourse increases the correlation with human judgments.