Evaluating Pronominal Anaphora in Machine Translation:An Evaluation Measure and a Test Suite
Table 1: Inter-annotator agreement.
['[BOLD] Language  [BOLD] Pair', '[BOLD] Nb. of  [BOLD] Ann.', '[BOLD] Avg%  [BOLD] Ref', '[BOLD] AC1  [BOLD] (Incl. Ties)', '[BOLD] AC1  [BOLD] (Excl. Ties)']
We asked the participants to choose the text with the better pronoun, their choices being candidate A, candidate B, equivalent translations (tie), “neither is correct”, and “invalid candidates” (the highlighted words are not pronouns or are the wrong pronouns due to misalignment). All examples marked as these last two were excluded from further consideration. Each participant annotated a total of 500 examples per language pair. We also report the proportion of cases where the participants preferred the reference translation over the noisy version (see Avg%Ref). We can see that there is high agreement for all language pairs, ranging from 0.82 to 0.89. The ties seem to be the major source of disagreement: excluding them yields agreements in the range of 0.91–0.97.