Evaluating Pronominal Anaphora in Machine Translation:An Evaluation Measure and a Test Suite
Table 5: Accuracy and AC1 Agreement for the ELMo-based model predictions on the study dataset.
['[BOLD] Language', '[BOLD] Acc.(ELMo)', '[BOLD] AC1 Agr.']
Finally, we test our ELMo - common reference context model on the held-out dataset used for the user study. The accuracy of the system is lower on the study dataset. Note that since the training data was filtered based on the pronoun pairs with high agreement from the study, the study dataset contains pronoun pairs with low agreement that were not seen during training. We also calculate the agreement with human judgments excluding ties since our model does not handle ties. The overall agreement with the human judgments remains high. The model scored the noisy version higher than the reference in about 19% of the cases. Of these, 46% were pronoun pairs that were not seen during training.