Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
Table 2: Ablation study of key components of Mucko.
['[BOLD] Method', '[BOLD] Method', '[BOLD] Overall Accuracy  [BOLD] top-1', '[BOLD] Overall Accuracy  [BOLD] top-3']
(1) In models ‘1-3’, we evaluate the influence of each layer of graphs on the performance. We observe that the top-1 accuracy of ‘1’ and ‘2’ respectively decreases by 1.1% and 3.94% compared with the full model, which indicates that both semantic and visual graphs are beneficial to provide valuable evidence for answer inference. Thereinto, the visual information has greater impact than the semantic part. When removing both semantic and visual graphs, ‘3’ results in a significant decrease. (2) In models ‘4-6’, we assess the effectiveness of the proposed cross-modal graph convolutions. ‘4’, ‘5’ and ‘6’ respectively replace the ‘Semantic-to-Fact Conv.’ in ‘2’, ‘Visual-to-Fact Conv.’ in ‘1’ and both in full model by concatenation, i.e. concatenating the mean pooling of all the semantic/visual node features with each entity feature. The performance decreases when replacing the convolution from either S-to-F or V-to-F, or both simultaneously, which proves the benefits of cross-modal convolution in gathering complementary evidence from different modalities. (3) We evaluate the influence of the relationships in the heterogeneous graph. We omit the relational features rij in all the three layers in ‘7’ and the performance decreases by nearly 1% on top-1 accuracy. It proves the benefits of relational information, though it is less influential than the modality information.