An Annotated Corpus of Reference Resolution for Interpreting Common Grounding
Table 6: Results of the reference resolution task grouped by the number of referents in the gold annotation (along with the average count of such markables in the test set).
['# Referents', '% Accuracy', '% Exact Match', 'Count']
To demonstrate the advantages of our approach for interpreting and analyzing dialogue systems, we give a more detailed analysis of TSEL-REF-DIAL model which performed well on all three tasks. In terms of the exact match rate, we found that the model performs very well on 0 and 7 referents: this is because most of them can be recognized at the superficial level, such as “none of them”, “all of mine” or “I don’t have that”. However, the model struggles on all other cases: the results are especially worse for markables with more than 1 referent. This shows that the model still lacks the ability of precisely tracking multiple referents, which can be expressed in complex, pragmatic ways (such as groupings).