A Revised Generative Evaluation of Visual Dialogue
Table 3: Evaluating the utility of automated reference sets Σ on standard VD evaluation. CCA-aq models were trained on the indicated subsets (Ht or all) of VisDial (v1.0), with answers from different sets (‘Ref’), and tested on the evaluation test server to compute standard metrics. Arrows indicate which direction is better.
['Train Set', 'Train #QA pairs', 'Ref', 'MR ↓', 'R@1 ↑', 'R@5 ↑', 'R@10 ↑', 'MRR ↑', 'NDCG ↑']
Finally, in (3), we intuit that if reference sets Σ contain answers similar to the correct answer, then a model trained on only these sets should improve performance on the \glsVD task. We, therefore, pair each question in the training subset Ht with each of the answers in its corresponding Σ, and train a \glsCCA-aq model. As a baseline, we repeat this experiment, but pairing the questions with answers from H instead of As a further check, we train a \glsCCA-aq model on Ht, but only between questions and their single ground-truth answers Agt (as opposed to all answers in H or Σ). Finally we conduct (3) across the whole dataset, learning a \glsCCA-aq model using Σ, over the entire training data of VisDial (v1.0). We observe a substantial improvement in \glsNDCG, with what is effectively a simple data augmentation procedure using Σ. This three-part verification supports the existence of valid answers in the automated reference set, which subsequently supports our revised evaluation scheme.