The PhotoBook Dataset:Building Common Ground through Visually-Grounded Dialogue
Table 1: Avg. token counts in COCO captions and the first and last descriptions in PhotoBook, plus their cosine distance to the caption’s cluster mean vector. The distance between first and last descriptions is 0.083.
['[BOLD] Source', '[BOLD] # Tokens', '[BOLD] # Content', '[BOLD] Distance']
In a small-scale pilot study, Ilinykh et al. We argue that in the PhotoBook task referring expressions are not only adapted based on the goal-oriented nature of the interaction but also by incorporating the developing common ground between the participants. This effect becomes most apparent when collecting all referring expressions for a specific target image produced during the different rounds of a game in its coreference chain. The following excerpt displays such a coreference chain extracted from the PhotoBook dataset: [itemsep=-1pt,leftmargin=14pt] A: Do you have a boy with a teal coloured shirt with yellow holding a bear with a red shirt? B: Boy with teal shirt and bear with red shirt? A: Teal shirt boy? To quantify the effect of referring expression refinement, we compare participants’ first and last descriptions to a given target image with the image’s captions provided in the MS COCO dataset. For this purpose we manually annotated the first and last expressions referring to a set of six target images across ten random games in the PhotoBook dataset. Before filtering, first referring expressions do not significantly differ in length from the COCO captions. Last descriptions however are significantly shorter than both the COCO captions and first descriptions. After filtering for content words, no significant differences remain. We also calculate the cosine distance between the three different descriptions based on their average word vectors. mean word vector, which is confirmed in our results. Comparing the distribution of word classes in the captions and referring expressions finally revealed a similar distribution in first referring expressions and COCO captions, and a significantly different distribution in last referring expressions, among other things doubling the relative frequency of nouns.