CohEval: Benchmarking Coherence Models
Table 5: Machine Translation setting results on WMT17-18 data. Accuracies: % of times reference scored higher and AC1 agreements for system translation rankings between annotators and models. Annotators: three native English speakers in a paid user study.
['[BOLD] Model', '[BOLD] Acc. (%)', '[BOLD] AC1 Agr.']
To validate the rankings of the translations given by the coherence models, we obtain rankings given by humans in a user study. One of the given translations is the reference, used as a control, and to validate our assumption that the reference is more coherent than the system translations. Participants chose the reference as more coherent with an agreement of 0.84, confirming our assumption and justifying our training setup. We see that the UnifiedModel which has the highest accuracy in scoring reference texts as higher, also has the highest agreement for rankings with humans. The rest of the models have similar agreements with human rankings. Surprisingly, the traditional EGrid model has high agreements with humans.