Analysing Coreference in Transformer Outputs
Table 3: BLEU and METEOR (MTR) scores for the 3 systems on our full test set (all) and the subset of sentences where coreference occurrs (coref). The number of erroneous mentions is shown for comparison.
['[EMPTY]', '[BOLD] news [ITALIC] all BLEU', '[BOLD] news [ITALIC] all MTR', '[BOLD] news [ITALIC] coref BLEU', '[BOLD] news [ITALIC] coref MTR', '#mention err.', '[BOLD] TED [ITALIC] all BLEU', '[BOLD] TED [ITALIC] all MTR', '[BOLD] TED [ITALIC] coref BLEU', '[BOLD] TED [ITALIC] coref MTR', '#mention err.']
We evaluate the quality of the three transformer engines with two automatic metrics, BLEU Papineni et al. For news, the best system is that trained on more data, S2; but for TED talks S3 with less data has the best performance. However, even if the systems differ in the overall performance, the change is not related to the number of errors in coreference chains. Whereas the number of errors correlates with translation quality (as measured by BLEU) for newscoref this is not the case of TEDcoref.