A Revised Generative Evaluation of Visual Dialogue
Table 8: Extended evaluation of the utility of automatic reference set construction methods (M,Σ,G) on the standard VD evaluation. Models were trained using CCA on the indicated subsets (Ht, all train, or all trainval) of VisDial v1.0, using answers drawn from different sets (‘Ref’), and tested on the evaluation test server.
['Train Set', 'Train #QA pairs', 'Ref', 'MR', 'R@1', 'R@5', 'R@10', 'MRR', 'NDCG']
We show that the \glsCCA-aq model, trained with the automatic reference sets rather than the human-annotated sets H achieve equal or better performance on the \glsVD task. We measure performance primarily by the \glsNDCG score. For completeness, we also show performance in the other rank-based metrics.