What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues
Table 1: Experimental results on VisPro. Precision (P), recall (R) and F1 scores are presented. The best performed F1 scores are indicated with bold font.
['Model', 'Discussed P', 'Discussed R', 'Discussed F1', 'Not Discussed P', 'Not Discussed R', 'Not Discussed F1', 'Overall P', 'Overall R', 'Overall F1']
Our proposed model VisCoref outperforms all the baseline models significantly, which indicates that the visual information is crucial for resolving pronouns in dialogues. Besides that, we also have the following interesting findings: [leftmargin=*] For all the conventional models, the “Not Discussed” pronouns whose antecedents are absent in dialogues are more challenging than “Discussed” pronouns whose antecedents appear in the dialogue context. The reason behind is that if the correct mentions appear in the near context of pronouns, the information about the correct mention can be aggregated to the targeting pronoun through either human-designed rules or deep neural networks (Bi-LSTM). However, when the correct mention is not available in the near context, it is quite challenging for conventional models to understand the dialogue and correctly ground the pronoun to the object both speakers can see, as they do not have the support of visual information. As is shown in the result of the “End-to-end+Visual” model, simply concatenating the visual feature to the contextual feature can help resolve “Discussed” pronouns but may hurt the performance of the model on “Not Discussed” pronouns. Different from them, the proposed Viscoref can improve the resolution of both the “Discussed” and “Not Discussed” pronouns. There are mainly two reasons behind: (1) The visual information in our model is first converted into textual labels and then transformed into vector representation in the same way as the dialogue context. Thus the vector space of contextual and visual information is perfectly aligned. (2) We introduce a hyper-parameter λvis to balance the influence of different knowledge resources. Even though our model outperforms all the baseline methods, we still can observe a huge gap between our model and human being. It indicates that current models still cannot fully understand the dialogue even with the support of visual information and further proves the value and necessity of proposing VisPro.