Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
Table A.3: Count of ambiguous examples in human static evaluation.
['[EMPTY]', 'Cornell HRED', 'Cornell VHRED', 'Cornell VHCR', 'Reddit HRED', 'Reddit VHRED', 'Reddit VHCR']
Noting the high disagreement between annotators in this task, we further examined the ambiguous examples in the human evaluation test set. We define an ambiguous example as a question where an equal number of annotators select the first sentence as better as the second sentence. If the two examples were similar, annotators would select the “tied" option. An equal number of selections for each answer as the winner indicates a disagreement in perception. The number of ambiguous samples further highlights the noisy and unreliable nature of static single-turn evaluation.