Evaluating Pronominal Anaphora in Machine Translation:An Evaluation Measure and a Test Suite
Table 4: Results on WMT14 with different contexts.
['[BOLD] Exp', '[BOLD] Context  [BOLD] Setting', '[BOLD] Test', '[BOLD] Acc.  [BOLD] (Glove)', '[BOLD] Acc.  [BOLD] (ELMo)']
The first three experiments in The results on the noisy data (Exp. 1-2) are indicative of how sensitive our model is to pronouns, since the difference between R and R^{\prime} is in a single pronoun. The results on the reference/system translation pair R and S (Exp. 3) is indicative of performance in a real use-case. In case of the GloVe-BiLSTM model, using attention instead of averaging pronouns does not help. However, using ELMo greatly improves the accuracy of the baseline and also leads to an improvement while using attention. ELMo can model syntactic and semantic information and improve co-reference resolution Peters:2018, which could be a contributing factor.