Reasoning Visual Dialogs with Structural and Partial Observations
Table 4: Ablation study of the key components of our methods on VisDial v0.9 dataset [9]. See subsection 4.4 for more details.
['Methods', 'MRR ↑', 'R@1 ↑', 'R@5 ↑', 'R@10 ↑', 'Mean ↓']
To assess the effect of some essential component of our model, we implement and test several variants: (i) constant graph that fixes edge weight between each pair of nodes to be 1; (ii) graph without EM iteration; and (iii) graph with n EM iterations. We summarize our observations here: (a) model without EM iterations performs the worst among all variants. This shows the importance of iteratively updating the node embeddings. (b) In our experiments, message passing with 3 iterations shows the best performance of our proposed model. (c) model using constant graph (3 iterations) performs better than worse than the model without EM iterations, since it allows iterative updates of node embeddings. However, it is outperformed by other iterative models with a dynamic structure, since all incoming messages are treated equally. This shows the importance of edge weights: they filter out misleading messages while allowing information flow.