Visual Dialog
Table 5: Performance of methods on VisDial v0.5, measured by mean reciprocal rank (MRR), recall@k for k={1,5,10} and mean rank. Note that higher is better for MRR and recall@k, while lower is better for mean rank. Memory Network has the best performance in both discriminative and generative settings.
['[EMPTY]', '[BOLD] Model', '[BOLD] MRR', '[BOLD] R@1', '[BOLD] R@5', '[BOLD] R@10', '[BOLD] Mean']
Results. Tab. Tab. A few key takeaways – First, as expected, all learning based models significantly outperform non-learning baselines. Second, all discriminative models significantly outperform generative models, which as we discussed is expected since discriminative models can tune to the biases in the answer options. This improvement comes with the significant limitation of not being able to actually generate responses, and we recommend the two decoders be viewed as separate use cases. Third, our best generative and discriminative models are MN-QIH-G with 0.44 MRR, and MN-QIH-D with 0.53 MRR that outperform a suite of models and sophisticated baselines. Fourth, we observe that models with H perform better than Q-only models, highlighting the importance of history in VisDial. Fifth, models looking at I outperform both the blind models (Q, QH) by at least 2% on recall@1 in both decoders. Finally, models that use both H and I have best performance.