Automatic Reference-Based Evaluation of Pronoun TranslationMisses the Point
Table 4: Common cases of disagreement for anaphoric, pleonastic, and event reference pronouns
['[BOLD] Category', '[BOLD] V', '[BOLD] E', '[BOLD] I', '[BOLD] O']
APT performs particularly poorly on the assessment of pronouns belonging to the anaphoric inter-sentential non-subject “it” and anaphoric singular “they” categories. 1) APT does not consider pronoun-antecedent head agreement so many valid alternative translations involving personal pronouns are marked as incompatible (case 3), but as correct by the human judges. 2) Substitutions between pronouns are governed by much more complex rules than the simple pronoun equivalence mechanism in APT suggests. 3) APT does not consider the use of impersonal pronouns such as c’ in place of the feminine personal pronoun elle or the plural forms ils and elles.