A Revised Generative Evaluation of Visual Dialogue
Table 5: Results for SOTA vs. CCA on the VisDial v0.9 and v1.0 dataset. CCA achieves comparable performance in mean rank (MR) while ignoring both image and dialogue sequence. Note, rpn (Ren et al., 2015).
['SOTA', '[ITALIC] v0.9', 'Model hciae-g-dis', 'I/QA features VGG-19/learned', 'MR  [BOLD] 14.23', 'R@1 44.35', 'R@5 65.28', 'R@10 71.55', 'MRR 0.5467', 'NDCG -']
This is consistent on Hv and the full validation set, despite Hv being 10-fold smaller—a further confirmation of Σ’s utility. Note, results across all metrics are well below the reference baselines ΓH/ΓΣ, indicating there is still room for improvement. Applying a beam-search on top of these models has the ability to further enrich the generations and improve performance on all metrics, as shown by hrea-qih-g *. It is expected that applying a beam search to the best-performer hciae-dis-g would yield similar improvements. This suggests that just because a model can rank a single-ground truth answer highest, does not necessarily make it the best generative \glsVD agent. Our suite of overlap metrics and embedding distance metrics may help to explain why. For example, cider n=1 is a proxy for how well a model performs on one-word answers, which are highly prevalent in the dataset (e.g. “Yes”/“No”). bert, on the other hand, may help to measure generations with the closest semantic similarity to the reference sets. Indeed, this is the sort of flexibility of purpose that is required when evaluating complex multi-modal tasks like \glsVD. We extend the work in Massiceti et al.