Visual Coreference Resolution in Visual Dialog using Neural Module Networks
Table 6: Retrieval performance on the validation set of VisDial v0.9 [13] (generative models). Higher the better for mean reciprocal rank (MRR) and recall@k (R@1, R@5, R@10), while lower the better for mean rank. Our CorefNMN model outperforms all other models in R@5, R@10, and mean rank. However, the neural module network baseline (NMN) has the best R@1, and perhaps as a result, the best MRR as well. *indicates numbers obtained from authors for models retrained on ResNet-152 features.
['[BOLD] Model', '[BOLD] MRR', '[BOLD] R@1', '[BOLD] R@5', '[BOLD] R@10', '[BOLD] Mean']
Results. We summarize our observations from Tab. : (a) Our CorefNMN outperforms all other approaches according to R@5, R@10, and mean rank metrics, highlighting the importance of explicitly resolving coreferences for visual dialog. Specifically, our mean rank of 15.69 is a 4% improvement over the NMN baseline. (b) However, the NMN baseline has a higher R@1, and perhaps as a result, the best MRR. A possible reason could be due to the noisy supervision from the out-of-domain, automatic, text-based coreference tool, as we entirely rely on it to predict Refer, the module responsible for coreference resolution. (c) Our novel way of handling captions using modules (indicated LauxC) boosts the full model, while the hurting the ablation without the memory augmentation. That is, CorefNMN∖Mem∖LauxC is better than CorefNMN∖Mem across all metrics, while CorefNMN∖LauxC is uniformly worse than CorefNMN.