Exploring Gap Filling as a Cheaper Alternative to Reading Comprehension Questionnaires when Evaluating Machine Translation for Gisting
Table 2: Effect in success rates of allowing for synonyms in GF
['System', 'GF scores with synonyms Overall', 'GF scores with synonyms 10%', 'GF scores with synonyms 20%', 'GF scores without synonyms Overall', 'GF scores without synonyms 10%', 'GF scores without synonyms 20%']
Allowing for synonyms: The GF success scores reported thus far have been computed by giving credit only to exact matches. We have studied giving credit to synonyms observed in informant work, namely to those appearing at least twice (in the work of all informants) that, according to one of the authors, preserved the meaning of the problem sentence, or were trivial spelling or case variations. A total of 124 frequent valid substitutions were considered. The relative ranking of MT systems is maintained; the statistical significance of the homebrew Moses results versus Bing results is maintained, and two additional statistically significant differences appear: Google vs. homebrew Moses and Systran vs. homebrew Moses. The statistical significance of the effect of gap density disappears when allowing for synonyms. This indicates that it would be beneficial to assign credit to synonyms if the necessary language resources are available or if further analysis of actual GF results is feasible.